JUnit version 4.13.2
.16:11:06.273 [INFO - main] (RandomUtils.java:49) Using random seed: 1786464666268
16:11:06.314 [INFO - main] (RetryRule.java:138) Creating RetryStatement with result reporter and retry count of 0 (no retries)
16:11:06.365 [INFO - main] (KuduTestHarness.java:155) Creating a new MiniKuduCluster...
16:11:06.378 [INFO - main] (TempDirUtils.java:71) Using the temp directory defined by TEST_TMPDIR: /tmp/dist-test-tasklXYFrv/test-tmp
16:11:06.380 [INFO - main] (KuduBinaryLocator.java:83) Using Kudu binary directory specified by system property 'kuduBinDir': ../../../build/tsan/bin
16:11:06.384 [INFO - main] (MiniKuduCluster.java:217) Starting process: [/tmp/dist-test-tasklXYFrv/java/build/dist-test/../../../build/tsan/bin/kudu, test, mini_cluster, --serialization=pb]
16:11:06.648 [DEBUG - main] (MiniKuduCluster.java:185) Request: create_cluster {
  num_masters: 3
  num_tservers: 3
  enable_kerberos: false
  cluster_root: "/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442"
  hms_mode: NONE
  mini_kdc_options {
  }
  principal: "kudu"
  mini_oidc_options {
  }
}

16:11:06.787 [DEBUG - main] (MiniKuduCluster.java:195) Response: 
16:11:06.788 [DEBUG - main] (MiniKuduCluster.java:185) Request: start_cluster {
}

16:11:06.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:06.792182 17322 mini_oidc.cc:152] Starting JWKS server
16:11:06.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:06.795961 17322 webserver.cc:460] Webserver started at http://127.0.0.1:45787/ using document root <none> and password file <none>
16:11:06.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:06.796380 17322 mini_oidc.cc:189] Starting OIDC Discovery server
16:11:06.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:06.798385 17322 webserver.cc:460] Webserver started at http://127.0.0.1:40049/ using document root <none> and password file <none>
16:11:06.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-08-11T16:11:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
16:11:06.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-08-11T16:11:06Z Disabled control of system clock
16:11:06.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:06.851292 17322 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu
16:11:06.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu
16:11:06.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/master-0/wal
16:11:06.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/master-0/data
16:11:06.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
16:11:06.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
16:11:06.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
16:11:06.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:11:06.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:11:06.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:11:06.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/master-0/logs
16:11:06.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/master-0/data/info.pb
16:11:06.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:11:06.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
16:11:06.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
16:11:06.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
16:11:06.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
16:11:06.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:11:06.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:11:06.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:11:06.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
16:11:06.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
16:11:06.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
16:11:06.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
16:11:06.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.234.190:33901
16:11:06.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.234.190
16:11:06.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
16:11:06.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.234.148:43519
16:11:06.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:11:06.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:11:06.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
16:11:06.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
16:11:06.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.16.234.190:33901,127.16.234.189:32987,127.16.234.188:38065
16:11:06.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
16:11:06.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:45787/jwks.json with env {}
16:11:07.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:07.149726 17347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
16:11:07.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:07.150400 17347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
16:11:07.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:07.151211 17347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
16:11:07.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:07.182106 17347 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
16:11:07.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:07.182410 17347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
16:11:07.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:07.182610 17347 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
16:11:07.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:07.182788 17347 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
16:11:07.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:07.182997 17347 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
16:11:07.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:07.183205 17347 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:45787/jwks.json
16:11:07.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:07.220238 17347 master_runner.cc:386] Master server non-default flags:
16:11:07.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:11:07.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.234.148:43519
16:11:07.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:11:07.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
16:11:07.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/master-0/data
16:11:07.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/master-0/wal
16:11:07.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
16:11:07.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.16.234.190:33901,127.16.234.189:32987,127.16.234.188:38065
16:11:07.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:11:07.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:11:07.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
16:11:07.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.234.190:33901
16:11:07.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
16:11:07.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
16:11:07.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
16:11:07.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:45787/jwks.json
16:11:07.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:11:07.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:11:07.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/master-0/data/info.pb
16:11:07.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.234.190
16:11:07.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
16:11:07.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
16:11:07.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:11:07.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
16:11:07.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
16:11:07.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:11:07.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/master-0/logs
16:11:07.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:11:07.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
16:11:07.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
16:11:07.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
16:11:07.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
16:11:07.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
16:11:07.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
16:11:07.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 15:12:52 UTC on ccycloud.quasar-tzrfpd.root.comops.site
16:11:07.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4580143
16:11:07.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
16:11:07.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:07.221525 17347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
16:11:07.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:07.223682 17347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
16:11:07.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:07.242357 17360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:11:08.341 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:07.242400 17358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:11:08.341 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:07.243918 17347 server_base.cc:1034] running on GCE node
16:11:08.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:07.243937 17364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:11:08.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:08.345433 17347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
16:11:08.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:08.348158 17347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
16:11:08.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:08.349543 17347 hybrid_clock.cc:648] HybridClock initialized: now 1786464668349511 us; error 52 us; skew 500 ppm
16:11:08.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:08.350467 17347 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
16:11:08.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:08.356817 17347 webserver.cc:460] Webserver started at http://127.16.234.190:40811/ using document root <none> and password file <none>
16:11:08.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:08.357740 17347 fs_manager.cc:359] Metadata directory not provided
16:11:08.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:08.357926 17347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
16:11:08.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:08.358376 17347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
16:11:08.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:08.363269 17347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/master-0/data/instance:
16:11:08.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "2ea4ce510c534787afa58a33455b3bff"
16:11:08.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 16:11:08 on dist-test-slave-zj8q"
16:11:08.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:08.364408 17347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/master-0/wal/instance:
16:11:08.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "2ea4ce510c534787afa58a33455b3bff"
16:11:08.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 16:11:08 on dist-test-slave-zj8q"
16:11:08.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:08.371528 17347 fs_manager.cc:691] Time spent creating directory manager: real 0.007s	user 0.008s	sys 0.000s
16:11:08.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:08.377280 17373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
16:11:08.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:08.378420 17347 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
16:11:08.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:08.378715 17347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/master-0/data,/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/master-0/wal
16:11:08.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "2ea4ce510c534787afa58a33455b3bff"
16:11:08.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 16:11:08 on dist-test-slave-zj8q"
16:11:08.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:08.379053 17347 fs_report.cc:383] FS layout report
16:11:08.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
16:11:08.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/master-0/wal
16:11:08.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/master-0/wal
16:11:08.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/master-0/data/data
16:11:08.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
16:11:08.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
16:11:08.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
16:11:08.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
16:11:08.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
16:11:08.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
16:11:08.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
16:11:08.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
16:11:08.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
16:11:08.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
16:11:08.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
16:11:08.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:08.410349 17347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
16:11:08.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:08.411793 17347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
16:11:08.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:08.412212 17347 kserver.cc:163] Server-wide thread pool size limit: 3276
16:11:08.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:08.481817 17347 rpc_server.cc:307] RPC server started. Bound to: 127.16.234.190:33901
16:11:08.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:08.481916 17470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.234.190:33901 every 8 connection(s)
16:11:08.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:08.484403 17347 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/master-0/data/info.pb
16:11:08.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:08.489379 17471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
16:11:08.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:08.492187 17322 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu as pid 17347
16:11:08.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:08.492568 17322 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/master-0/wal/instance
16:11:08.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:08.495900 17322 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu
16:11:08.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu
16:11:08.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/master-1/wal
16:11:08.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/master-1/data
16:11:08.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
16:11:08.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
16:11:08.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
16:11:08.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:11:08.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:11:08.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:11:08.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/master-1/logs
16:11:08.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/master-1/data/info.pb
16:11:08.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:11:08.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
16:11:08.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
16:11:08.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
16:11:08.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
16:11:08.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:11:08.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:11:08.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:11:08.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
16:11:08.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
16:11:08.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
16:11:08.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
16:11:08.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.234.189:32987
16:11:08.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.234.189
16:11:08.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
16:11:08.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.234.148:43519
16:11:08.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:11:08.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:11:08.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
16:11:08.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
16:11:08.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.16.234.190:33901,127.16.234.189:32987,127.16.234.188:38065
16:11:08.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
16:11:08.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:45787/jwks.json with env {}
16:11:08.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:08.502394 17471 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.234.190" port: 33901 } has no permanent_uuid. Determining permanent_uuid...
16:11:08.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:08.531474 17471 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.234.189" port: 32987 } has no permanent_uuid. Determining permanent_uuid...
16:11:08.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:08.534508 17377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.234.189:32987: connect: Connection refused (error 111)
16:11:08.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:08.538467 17471 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.234.189:32987: Network error: Client connection negotiation failed: client connection to 127.16.234.189:32987: connect: Connection refused (error 111)
16:11:08.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:08.588006 17471 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.234.189" port: 32987 } attempt: 1
16:11:08.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:08.591693 17471 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.234.189:32987: Network error: Client connection negotiation failed: client connection to 127.16.234.189:32987: connect: Connection refused (error 111)
16:11:08.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:08.660177 17471 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.234.189" port: 32987 } attempt: 2
16:11:08.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:08.663961 17471 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.234.189:32987: Network error: Client connection negotiation failed: client connection to 127.16.234.189:32987: connect: Connection refused (error 111)
16:11:08.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:08.755404 17471 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.234.189" port: 32987 } attempt: 3
16:11:08.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:08.759503 17471 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.234.189:32987: Network error: Client connection negotiation failed: client connection to 127.16.234.189:32987: connect: Connection refused (error 111)
16:11:08.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:08.801764 17473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
16:11:08.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:08.802369 17473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
16:11:08.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:08.802889 17473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
16:11:08.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:08.833504 17473 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
16:11:08.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:08.833779 17473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
16:11:08.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:08.833968 17473 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
16:11:08.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:08.834146 17473 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
16:11:08.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:08.834367 17473 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
16:11:08.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:08.834508 17473 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:45787/jwks.json
16:11:08.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:08.868993 17473 master_runner.cc:386] Master server non-default flags:
16:11:08.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:11:08.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.234.148:43519
16:11:08.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:11:08.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
16:11:08.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/master-1/data
16:11:08.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/master-1/wal
16:11:08.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
16:11:08.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.16.234.190:33901,127.16.234.189:32987,127.16.234.188:38065
16:11:08.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:11:08.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:11:08.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
16:11:08.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.234.189:32987
16:11:08.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
16:11:08.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
16:11:08.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
16:11:08.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:45787/jwks.json
16:11:08.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:11:08.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:11:08.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/master-1/data/info.pb
16:11:08.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.234.189
16:11:08.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
16:11:08.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
16:11:08.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:11:08.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
16:11:08.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
16:11:08.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:11:08.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/master-1/logs
16:11:08.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:11:08.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
16:11:08.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
16:11:08.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
16:11:08.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
16:11:08.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
16:11:08.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
16:11:08.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 15:12:52 UTC on ccycloud.quasar-tzrfpd.root.comops.site
16:11:08.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4580143
16:11:08.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
16:11:08.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:08.870276 17473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
16:11:08.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:08.871824 17473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
16:11:08.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:08.889313 17496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:11:08.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:08.902885 17471 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.234.189" port: 32987 } attempt: 4
16:11:08.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:08.906332 17471 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.234.189:32987: Network error: Client connection negotiation failed: client connection to 127.16.234.189:32987: connect: Connection refused (error 111)
16:11:09.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:09.205713 17471 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.234.189" port: 32987 } attempt: 5
16:11:09.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:09.209676 17471 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.234.189:32987: Network error: Client connection negotiation failed: client connection to 127.16.234.189:32987: connect: Connection refused (error 111)
16:11:09.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:09.757103 17471 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.234.189" port: 32987 } attempt: 6
16:11:09.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:09.764065 17471 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.234.189:32987: Network error: Client connection negotiation failed: client connection to 127.16.234.189:32987: connect: Connection refused (error 111)
16:11:09.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:08.889348 17500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:11:09.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:08.891309 17473 server_base.cc:1034] running on GCE node
16:11:09.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:08.894647 17493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:11:09.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:09.995312 17473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
16:11:09.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:09.998464 17473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
16:11:10.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:09.999895 17473 hybrid_clock.cc:648] HybridClock initialized: now 1786464669999852 us; error 63 us; skew 500 ppm
16:11:10.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:10.000756 17473 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
16:11:10.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:10.010767 17473 webserver.cc:460] Webserver started at http://127.16.234.189:33193/ using document root <none> and password file <none>
16:11:10.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:10.011860 17473 fs_manager.cc:359] Metadata directory not provided
16:11:10.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:10.012059 17473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
16:11:10.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:10.012455 17473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
16:11:10.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:10.019172 17473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/master-1/data/instance:
16:11:10.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "5eaadf6df4764214a317ceb58603922b"
16:11:10.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 16:11:10 on dist-test-slave-zj8q"
16:11:10.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:10.020452 17473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/master-1/wal/instance:
16:11:10.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "5eaadf6df4764214a317ceb58603922b"
16:11:10.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 16:11:10 on dist-test-slave-zj8q"
16:11:10.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:10.027863 17473 fs_manager.cc:691] Time spent creating directory manager: real 0.007s	user 0.004s	sys 0.004s
16:11:10.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:10.033495 17512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
16:11:10.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:10.034615 17473 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
16:11:10.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:10.034919 17473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/master-1/data,/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/master-1/wal
16:11:10.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "5eaadf6df4764214a317ceb58603922b"
16:11:10.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 16:11:10 on dist-test-slave-zj8q"
16:11:10.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:10.035245 17473 fs_report.cc:383] FS layout report
16:11:10.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
16:11:10.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/master-1/wal
16:11:10.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/master-1/wal
16:11:10.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/master-1/data/data
16:11:10.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
16:11:10.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
16:11:10.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
16:11:10.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
16:11:10.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
16:11:10.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
16:11:10.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
16:11:10.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
16:11:10.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
16:11:10.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
16:11:10.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
16:11:10.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:10.066473 17473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
16:11:10.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:10.067950 17473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
16:11:10.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:10.068404 17473 kserver.cc:163] Server-wide thread pool size limit: 3276
16:11:10.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:10.138186 17473 rpc_server.cc:307] RPC server started. Bound to: 127.16.234.189:32987
16:11:10.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:10.138311 17593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.234.189:32987 every 8 connection(s)
16:11:10.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:10.140872 17473 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/master-1/data/info.pb
16:11:10.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:10.146036 17594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
16:11:10.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:10.146077 17322 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu as pid 17473
16:11:10.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:10.146452 17322 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/master-1/wal/instance
16:11:10.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:10.148397 17322 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu
16:11:10.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu
16:11:10.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/master-2/wal
16:11:10.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/master-2/data
16:11:10.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
16:11:10.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
16:11:10.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
16:11:10.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:11:10.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:11:10.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:11:10.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/master-2/logs
16:11:10.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/master-2/data/info.pb
16:11:10.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:11:10.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
16:11:10.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
16:11:10.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
16:11:10.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
16:11:10.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:11:10.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:11:10.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:11:10.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
16:11:10.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
16:11:10.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
16:11:10.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
16:11:10.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.234.188:38065
16:11:10.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.234.188
16:11:10.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
16:11:10.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.234.148:43519
16:11:10.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:11:10.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:11:10.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
16:11:10.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
16:11:10.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.16.234.190:33901,127.16.234.189:32987,127.16.234.188:38065
16:11:10.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
16:11:10.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:45787/jwks.json with env {}
16:11:10.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:10.159073 17594 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.234.190" port: 33901 } has no permanent_uuid. Determining permanent_uuid...
16:11:10.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:10.183125 17594 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.234.189" port: 32987 } has no permanent_uuid. Determining permanent_uuid...
16:11:10.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:10.198009 17594 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.234.188" port: 38065 } has no permanent_uuid. Determining permanent_uuid...
16:11:10.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:10.200230 17514 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.234.188:38065: connect: Connection refused (error 111)
16:11:10.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:10.204262 17594 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.234.188:38065: Network error: Client connection negotiation failed: client connection to 127.16.234.188:38065: connect: Connection refused (error 111)
16:11:10.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:10.253810 17594 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.234.188" port: 38065 } attempt: 1
16:11:10.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:10.257468 17594 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.234.188:38065: Network error: Client connection negotiation failed: client connection to 127.16.234.188:38065: connect: Connection refused (error 111)
16:11:10.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:10.326030 17594 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.234.188" port: 38065 } attempt: 2
16:11:10.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:10.329844 17594 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.234.188:38065: Network error: Client connection negotiation failed: client connection to 127.16.234.188:38065: connect: Connection refused (error 111)
16:11:10.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:10.421252 17594 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.234.188" port: 38065 } attempt: 3
16:11:10.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:10.424919 17594 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.234.188:38065: Network error: Client connection negotiation failed: client connection to 127.16.234.188:38065: connect: Connection refused (error 111)
16:11:10.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:10.465622 17596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
16:11:10.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:10.466230 17596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
16:11:10.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:10.466727 17596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
16:11:10.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:10.511013 17596 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
16:11:10.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:10.511294 17596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
16:11:10.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:10.511485 17596 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
16:11:10.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:10.511731 17596 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
16:11:10.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:10.511932 17596 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
16:11:10.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:10.512135 17596 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:45787/jwks.json
16:11:10.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:10.546916 17596 master_runner.cc:386] Master server non-default flags:
16:11:10.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:11:10.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.234.148:43519
16:11:10.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:11:10.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
16:11:10.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/master-2/data
16:11:10.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/master-2/wal
16:11:10.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
16:11:10.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.16.234.190:33901,127.16.234.189:32987,127.16.234.188:38065
16:11:10.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:11:10.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:11:10.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
16:11:10.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.234.188:38065
16:11:10.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
16:11:10.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
16:11:10.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
16:11:10.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:45787/jwks.json
16:11:10.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:11:10.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:11:10.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/master-2/data/info.pb
16:11:10.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.234.188
16:11:10.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
16:11:10.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
16:11:10.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:11:10.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
16:11:10.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
16:11:10.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:11:10.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/master-2/logs
16:11:10.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:11:10.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
16:11:10.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
16:11:10.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
16:11:10.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
16:11:10.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
16:11:10.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
16:11:10.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 15:12:52 UTC on ccycloud.quasar-tzrfpd.root.comops.site
16:11:10.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4580143
16:11:10.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
16:11:10.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:10.548235 17596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
16:11:10.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:10.550417 17596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
16:11:10.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:10.568288 17594 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.234.188" port: 38065 } attempt: 4
16:11:10.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:10.568951 17615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:11:10.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:10.572535 17594 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.234.188:38065: Network error: Client connection negotiation failed: client connection to 127.16.234.188:38065: connect: Connection refused (error 111)
16:11:10.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:10.824479 17471 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.234.189" port: 32987 } attempt: 7
16:11:10.839 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:10.838593 17471 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.234.188" port: 38065 } has no permanent_uuid. Determining permanent_uuid...
16:11:10.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:10.843948 17471 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.234.188:38065: Network error: Client connection negotiation failed: client connection to 127.16.234.188:38065: connect: Connection refused (error 111)
16:11:10.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:10.871909 17594 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.234.188" port: 38065 } attempt: 5
16:11:10.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:10.875424 17594 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.234.188:38065: Network error: Client connection negotiation failed: client connection to 127.16.234.188:38065: connect: Connection refused (error 111)
16:11:10.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:10.902318 17471 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.234.188" port: 38065 } attempt: 1
16:11:10.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:10.906047 17471 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.234.188:38065: Network error: Client connection negotiation failed: client connection to 127.16.234.188:38065: connect: Connection refused (error 111)
16:11:10.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:10.987475 17471 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.234.188" port: 38065 } attempt: 2
16:11:10.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:10.991545 17471 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.234.188:38065: Network error: Client connection negotiation failed: client connection to 127.16.234.188:38065: connect: Connection refused (error 111)
16:11:11.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:11.076943 17471 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.234.188" port: 38065 } attempt: 3
16:11:11.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:11.080842 17471 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.234.188:38065: Network error: Client connection negotiation failed: client connection to 127.16.234.188:38065: connect: Connection refused (error 111)
16:11:11.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:11.221246 17471 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.234.188" port: 38065 } attempt: 4
16:11:11.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:11.225860 17471 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.234.188:38065: Network error: Client connection negotiation failed: client connection to 127.16.234.188:38065: connect: Connection refused (error 111)
16:11:11.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:11.422974 17594 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.234.188" port: 38065 } attempt: 6
16:11:11.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:11.429893 17594 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.234.188:38065: Network error: Client connection negotiation failed: client connection to 127.16.234.188:38065: connect: Connection refused (error 111)
16:11:11.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:11.509369 17471 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.234.188" port: 38065 } attempt: 5
16:11:11.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:11.513175 17471 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.234.188:38065: Network error: Client connection negotiation failed: client connection to 127.16.234.188:38065: connect: Connection refused (error 111)
16:11:11.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:10.569839 17619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:11:11.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:10.570917 17616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:11:11.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:10.571570 17596 server_base.cc:1034] running on GCE node
16:11:11.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:11.689411 17596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
16:11:11.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:11.692565 17596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
16:11:11.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:11.693974 17596 hybrid_clock.cc:648] HybridClock initialized: now 1786464671693917 us; error 73 us; skew 500 ppm
16:11:11.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:11.694809 17596 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
16:11:11.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:11.701357 17596 webserver.cc:460] Webserver started at http://127.16.234.188:38981/ using document root <none> and password file <none>
16:11:11.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:11.702333 17596 fs_manager.cc:359] Metadata directory not provided
16:11:11.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:11.702605 17596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
16:11:11.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:11.703145 17596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
16:11:11.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:11.707947 17596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/master-2/data/instance:
16:11:11.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "cdd30a1bb4e2432e9b560c791116b727"
16:11:11.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 16:11:11 on dist-test-slave-zj8q"
16:11:11.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:11.708971 17596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/master-2/wal/instance:
16:11:11.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "cdd30a1bb4e2432e9b560c791116b727"
16:11:11.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 16:11:11 on dist-test-slave-zj8q"
16:11:11.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:11.715855 17596 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.001s	sys 0.008s
16:11:11.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:11.721266 17641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
16:11:11.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:11.722362 17596 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
16:11:11.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:11.722692 17596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/master-2/data,/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/master-2/wal
16:11:11.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "cdd30a1bb4e2432e9b560c791116b727"
16:11:11.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 16:11:11 on dist-test-slave-zj8q"
16:11:11.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:11.723059 17596 fs_report.cc:383] FS layout report
16:11:11.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
16:11:11.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/master-2/wal
16:11:11.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/master-2/wal
16:11:11.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/master-2/data/data
16:11:11.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
16:11:11.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
16:11:11.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
16:11:11.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
16:11:11.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
16:11:11.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
16:11:11.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
16:11:11.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
16:11:11.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
16:11:11.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
16:11:11.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
16:11:11.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:11.773996 17596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
16:11:11.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:11.775606 17596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
16:11:11.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:11.776124 17596 kserver.cc:163] Server-wide thread pool size limit: 3276
16:11:11.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:11.845418 17596 rpc_server.cc:307] RPC server started. Bound to: 127.16.234.188:38065
16:11:11.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:11.845511 17710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.234.188:38065 every 8 connection(s)
16:11:11.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:11.848228 17596 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/master-2/data/info.pb
16:11:11.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:11.849758 17322 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu as pid 17596
16:11:11.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:11.850268 17322 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/master-2/wal/instance
16:11:11.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:11.854754 17711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
16:11:11.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:11.872643 17711 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.234.190" port: 33901 } has no permanent_uuid. Determining permanent_uuid...
16:11:11.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:11.903415 17711 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.234.189" port: 32987 } has no permanent_uuid. Determining permanent_uuid...
16:11:11.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:11.913734 17711 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.234.188" port: 38065 } has no permanent_uuid. Determining permanent_uuid...
16:11:11.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:11.930409 17711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdd30a1bb4e2432e9b560c791116b727: Bootstrap starting.
16:11:11.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:11.936401 17711 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cdd30a1bb4e2432e9b560c791116b727: Neither blocks nor log segments found. Creating new log.
16:11:11.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:11.938638 17711 log.cc:824] T 00000000000000000000000000000000 P cdd30a1bb4e2432e9b560c791116b727: Log is configured to *not* fsync() on all Append() calls
16:11:11.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:11.942965 17711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdd30a1bb4e2432e9b560c791116b727: No bootstrap required, opened a new log
16:11:11.964 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:11.963706 17711 raft_consensus.cc:348] T 00000000000000000000000000000000 P cdd30a1bb4e2432e9b560c791116b727 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ea4ce510c534787afa58a33455b3bff" member_type: VOTER last_known_addr { host: "127.16.234.190" port: 33901 } } peers { permanent_uuid: "5eaadf6df4764214a317ceb58603922b" member_type: VOTER last_known_addr { host: "127.16.234.189" port: 32987 } } peers { permanent_uuid: "cdd30a1bb4e2432e9b560c791116b727" member_type: VOTER last_known_addr { host: "127.16.234.188" port: 38065 } }
16:11:11.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:11.964313 17711 raft_consensus.cc:374] T 00000000000000000000000000000000 P cdd30a1bb4e2432e9b560c791116b727 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
16:11:11.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:11.964496 17711 raft_consensus.cc:729] T 00000000000000000000000000000000 P cdd30a1bb4e2432e9b560c791116b727 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdd30a1bb4e2432e9b560c791116b727, State: Initialized, Role: FOLLOWER
16:11:11.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:11.966220 17711 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdd30a1bb4e2432e9b560c791116b727 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ea4ce510c534787afa58a33455b3bff" member_type: VOTER last_known_addr { host: "127.16.234.190" port: 33901 } } peers { permanent_uuid: "5eaadf6df4764214a317ceb58603922b" member_type: VOTER last_known_addr { host: "127.16.234.189" port: 32987 } } peers { permanent_uuid: "cdd30a1bb4e2432e9b560c791116b727" member_type: VOTER last_known_addr { host: "127.16.234.188" port: 38065 } }
16:11:11.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:11.969517 17723 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdd30a1bb4e2432e9b560c791116b727 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ea4ce510c534787afa58a33455b3bff" member_type: VOTER last_known_addr { host: "127.16.234.190" port: 33901 } } peers { permanent_uuid: "5eaadf6df4764214a317ceb58603922b" member_type: VOTER last_known_addr { host: "127.16.234.189" port: 32987 } } peers { permanent_uuid: "cdd30a1bb4e2432e9b560c791116b727" member_type: VOTER last_known_addr { host: "127.16.234.188" port: 38065 } } }
16:11:11.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:11.970412 17723 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdd30a1bb4e2432e9b560c791116b727 [sys.catalog]: This master's current role is: FOLLOWER
16:11:11.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:11.971361 17711 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdd30a1bb4e2432e9b560c791116b727 [sys.catalog]: configured and running, proceeding with master startup.
16:11:11.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:11.994124 17743 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cdd30a1bb4e2432e9b560c791116b727: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
16:11:11.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:11.994453 17743 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
16:11:12.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:11.990814 17596 master_runner.cc:186] Error getting master registration for 127.16.234.190:33901: OK, instance_id { permanent_uuid: "2ea4ce510c534787afa58a33455b3bff" instance_seqno: 1786464668462968 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
16:11:12.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.023424 17596 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
16:11:12.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:12.031684 17648 tablet.cc:2369] T 00000000000000000000000000000000 P cdd30a1bb4e2432e9b560c791116b727: Can't schedule compaction. Clean time has not been advanced past its initial value.
16:11:12.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.065627 17471 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.234.188" port: 38065 } attempt: 6
16:11:12.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.084612 17471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ea4ce510c534787afa58a33455b3bff: Bootstrap starting.
16:11:12.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.089710 17471 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ea4ce510c534787afa58a33455b3bff: Neither blocks nor log segments found. Creating new log.
16:11:12.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.091475 17471 log.cc:824] T 00000000000000000000000000000000 P 2ea4ce510c534787afa58a33455b3bff: Log is configured to *not* fsync() on all Append() calls
16:11:12.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.096298 17471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ea4ce510c534787afa58a33455b3bff: No bootstrap required, opened a new log
16:11:12.117 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.116671 17471 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ea4ce510c534787afa58a33455b3bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ea4ce510c534787afa58a33455b3bff" member_type: VOTER last_known_addr { host: "127.16.234.190" port: 33901 } } peers { permanent_uuid: "5eaadf6df4764214a317ceb58603922b" member_type: VOTER last_known_addr { host: "127.16.234.189" port: 32987 } } peers { permanent_uuid: "cdd30a1bb4e2432e9b560c791116b727" member_type: VOTER last_known_addr { host: "127.16.234.188" port: 38065 } }
16:11:12.117 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.117612 17471 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ea4ce510c534787afa58a33455b3bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
16:11:12.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.117947 17471 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ea4ce510c534787afa58a33455b3bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ea4ce510c534787afa58a33455b3bff, State: Initialized, Role: FOLLOWER
16:11:12.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.118902 17471 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ea4ce510c534787afa58a33455b3bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ea4ce510c534787afa58a33455b3bff" member_type: VOTER last_known_addr { host: "127.16.234.190" port: 33901 } } peers { permanent_uuid: "5eaadf6df4764214a317ceb58603922b" member_type: VOTER last_known_addr { host: "127.16.234.189" port: 32987 } } peers { permanent_uuid: "cdd30a1bb4e2432e9b560c791116b727" member_type: VOTER last_known_addr { host: "127.16.234.188" port: 38065 } }
16:11:12.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.122709 17748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ea4ce510c534787afa58a33455b3bff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ea4ce510c534787afa58a33455b3bff" member_type: VOTER last_known_addr { host: "127.16.234.190" port: 33901 } } peers { permanent_uuid: "5eaadf6df4764214a317ceb58603922b" member_type: VOTER last_known_addr { host: "127.16.234.189" port: 32987 } } peers { permanent_uuid: "cdd30a1bb4e2432e9b560c791116b727" member_type: VOTER last_known_addr { host: "127.16.234.188" port: 38065 } } }
16:11:12.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.123575 17748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ea4ce510c534787afa58a33455b3bff [sys.catalog]: This master's current role is: FOLLOWER
16:11:12.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.124390 17471 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ea4ce510c534787afa58a33455b3bff [sys.catalog]: configured and running, proceeding with master startup.
16:11:12.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:12.170668 17387 tablet.cc:2369] T 00000000000000000000000000000000 P 2ea4ce510c534787afa58a33455b3bff: Can't schedule compaction. Clean time has not been advanced past its initial value.
16:11:12.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:12.176406 17776 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2ea4ce510c534787afa58a33455b3bff: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
16:11:12.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:12.177177 17776 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
16:11:12.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.187238 17723 raft_consensus.cc:482] T 00000000000000000000000000000000 P cdd30a1bb4e2432e9b560c791116b727 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
16:11:12.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.187801 17723 raft_consensus.cc:504] T 00000000000000000000000000000000 P cdd30a1bb4e2432e9b560c791116b727 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ea4ce510c534787afa58a33455b3bff" member_type: VOTER last_known_addr { host: "127.16.234.190" port: 33901 } } peers { permanent_uuid: "5eaadf6df4764214a317ceb58603922b" member_type: VOTER last_known_addr { host: "127.16.234.189" port: 32987 } } peers { permanent_uuid: "cdd30a1bb4e2432e9b560c791116b727" member_type: VOTER last_known_addr { host: "127.16.234.188" port: 38065 } }
16:11:12.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.161288 17347 master_runner.cc:186] Error getting master registration for 127.16.234.189:32987: OK, instance_id { permanent_uuid: "5eaadf6df4764214a317ceb58603922b" instance_seqno: 1786464670119042 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
16:11:12.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.190542 17347 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
16:11:12.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.191730 17428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cdd30a1bb4e2432e9b560c791116b727" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ea4ce510c534787afa58a33455b3bff" is_pre_election: true
16:11:12.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.192674 17428 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2ea4ce510c534787afa58a33455b3bff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdd30a1bb4e2432e9b560c791116b727 in term 0.
16:11:12.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.198470 17645 leader_election.cc:302] T 00000000000000000000000000000000 P cdd30a1bb4e2432e9b560c791116b727 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ea4ce510c534787afa58a33455b3bff, cdd30a1bb4e2432e9b560c791116b727; no voters: 
16:11:12.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.200340 17723 leader_election.cc:288] T 00000000000000000000000000000000 P cdd30a1bb4e2432e9b560c791116b727 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ea4ce510c534787afa58a33455b3bff (127.16.234.190:33901), 5eaadf6df4764214a317ceb58603922b (127.16.234.189:32987)
16:11:12.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.200879 17723 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cdd30a1bb4e2432e9b560c791116b727 [term 0 FOLLOWER]: Leader pre-election won for term 1
16:11:12.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.201228 17723 raft_consensus.cc:482] T 00000000000000000000000000000000 P cdd30a1bb4e2432e9b560c791116b727 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
16:11:12.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.201524 17723 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdd30a1bb4e2432e9b560c791116b727 [term 0 FOLLOWER]: Advancing to term 1
16:11:12.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.208964 17723 raft_consensus.cc:504] T 00000000000000000000000000000000 P cdd30a1bb4e2432e9b560c791116b727 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ea4ce510c534787afa58a33455b3bff" member_type: VOTER last_known_addr { host: "127.16.234.190" port: 33901 } } peers { permanent_uuid: "5eaadf6df4764214a317ceb58603922b" member_type: VOTER last_known_addr { host: "127.16.234.189" port: 32987 } } peers { permanent_uuid: "cdd30a1bb4e2432e9b560c791116b727" member_type: VOTER last_known_addr { host: "127.16.234.188" port: 38065 } }
16:11:12.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.211520 17723 leader_election.cc:288] T 00000000000000000000000000000000 P cdd30a1bb4e2432e9b560c791116b727 [CANDIDATE]: Term 1 election: Requested vote from peers 2ea4ce510c534787afa58a33455b3bff (127.16.234.190:33901), 5eaadf6df4764214a317ceb58603922b (127.16.234.189:32987)
16:11:12.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.214068 17428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cdd30a1bb4e2432e9b560c791116b727" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ea4ce510c534787afa58a33455b3bff"
16:11:12.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.214682 17428 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ea4ce510c534787afa58a33455b3bff [term 0 FOLLOWER]: Advancing to term 1
16:11:12.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.221976 17428 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2ea4ce510c534787afa58a33455b3bff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdd30a1bb4e2432e9b560c791116b727 in term 1.
16:11:12.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.223912 17645 leader_election.cc:302] T 00000000000000000000000000000000 P cdd30a1bb4e2432e9b560c791116b727 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ea4ce510c534787afa58a33455b3bff, cdd30a1bb4e2432e9b560c791116b727; no voters: 
16:11:12.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.227077 17778 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cdd30a1bb4e2432e9b560c791116b727 [term 1 FOLLOWER]: Leader election won for term 1
16:11:12.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.232299 17778 raft_consensus.cc:686] T 00000000000000000000000000000000 P cdd30a1bb4e2432e9b560c791116b727 [term 1 LEADER]: Becoming Leader. State: Replica: cdd30a1bb4e2432e9b560c791116b727, State: Running, Role: LEADER
16:11:12.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.233808 17778 consensus_queue.cc:237] T 00000000000000000000000000000000 P cdd30a1bb4e2432e9b560c791116b727 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ea4ce510c534787afa58a33455b3bff" member_type: VOTER last_known_addr { host: "127.16.234.190" port: 33901 } } peers { permanent_uuid: "5eaadf6df4764214a317ceb58603922b" member_type: VOTER last_known_addr { host: "127.16.234.189" port: 32987 } } peers { permanent_uuid: "cdd30a1bb4e2432e9b560c791116b727" member_type: VOTER last_known_addr { host: "127.16.234.188" port: 38065 } }
16:11:12.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.246790 17723 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdd30a1bb4e2432e9b560c791116b727 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdd30a1bb4e2432e9b560c791116b727. Latest consensus state: current_term: 1 leader_uuid: "cdd30a1bb4e2432e9b560c791116b727" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ea4ce510c534787afa58a33455b3bff" member_type: VOTER last_known_addr { host: "127.16.234.190" port: 33901 } } peers { permanent_uuid: "5eaadf6df4764214a317ceb58603922b" member_type: VOTER last_known_addr { host: "127.16.234.189" port: 32987 } } peers { permanent_uuid: "cdd30a1bb4e2432e9b560c791116b727" member_type: VOTER last_known_addr { host: "127.16.234.188" port: 38065 } } }
16:11:12.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.250615 17723 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdd30a1bb4e2432e9b560c791116b727 [sys.catalog]: This master's current role is: LEADER
16:11:12.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.254338 17781 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
16:11:12.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.261469 17781 catalog_manager.cc:1484] Initializing Kudu cluster ID...
16:11:12.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.290460 17428 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2ea4ce510c534787afa58a33455b3bff [term 1 FOLLOWER]: Refusing update from remote peer cdd30a1bb4e2432e9b560c791116b727: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
16:11:12.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.294581 17778 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cdd30a1bb4e2432e9b560c791116b727 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ea4ce510c534787afa58a33455b3bff" member_type: VOTER last_known_addr { host: "127.16.234.190" port: 33901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
16:11:12.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.328557 17782 mvcc.cc:205] Tried to move back new op lower bound from 7317359297678983168 to 7317359297485729792. Current Snapshot: MvccSnapshot[applied={T|T < 7317359297678983168}]
16:11:12.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.332803 17781 catalog_manager.cc:1347] Generated new cluster ID: b784a3075a744d64807f5591c6b86af3
16:11:12.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.333117 17781 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
16:11:12.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.332930 17723 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdd30a1bb4e2432e9b560c791116b727 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cdd30a1bb4e2432e9b560c791116b727" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ea4ce510c534787afa58a33455b3bff" member_type: VOTER last_known_addr { host: "127.16.234.190" port: 33901 } } peers { permanent_uuid: "5eaadf6df4764214a317ceb58603922b" member_type: VOTER last_known_addr { host: "127.16.234.189" port: 32987 } } peers { permanent_uuid: "cdd30a1bb4e2432e9b560c791116b727" member_type: VOTER last_known_addr { host: "127.16.234.188" port: 38065 } } }
16:11:12.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.333722 17723 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdd30a1bb4e2432e9b560c791116b727 [sys.catalog]: This master's current role is: LEADER
16:11:12.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.340363 17748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ea4ce510c534787afa58a33455b3bff [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdd30a1bb4e2432e9b560c791116b727. Latest consensus state: current_term: 1 leader_uuid: "cdd30a1bb4e2432e9b560c791116b727" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ea4ce510c534787afa58a33455b3bff" member_type: VOTER last_known_addr { host: "127.16.234.190" port: 33901 } } peers { permanent_uuid: "5eaadf6df4764214a317ceb58603922b" member_type: VOTER last_known_addr { host: "127.16.234.189" port: 32987 } } peers { permanent_uuid: "cdd30a1bb4e2432e9b560c791116b727" member_type: VOTER last_known_addr { host: "127.16.234.188" port: 38065 } } }
16:11:12.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.343883 17748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ea4ce510c534787afa58a33455b3bff [sys.catalog]: This master's current role is: FOLLOWER
16:11:12.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.371096 17748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ea4ce510c534787afa58a33455b3bff [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cdd30a1bb4e2432e9b560c791116b727" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ea4ce510c534787afa58a33455b3bff" member_type: VOTER last_known_addr { host: "127.16.234.190" port: 33901 } } peers { permanent_uuid: "5eaadf6df4764214a317ceb58603922b" member_type: VOTER last_known_addr { host: "127.16.234.189" port: 32987 } } peers { permanent_uuid: "cdd30a1bb4e2432e9b560c791116b727" member_type: VOTER last_known_addr { host: "127.16.234.188" port: 38065 } } }
16:11:12.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.371868 17748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ea4ce510c534787afa58a33455b3bff [sys.catalog]: This master's current role is: FOLLOWER
16:11:12.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.383368 17781 catalog_manager.cc:1370] Generated new certificate authority record
16:11:12.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.386878 17781 catalog_manager.cc:1504] Loading token signing keys...
16:11:12.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.433957 17781 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cdd30a1bb4e2432e9b560c791116b727: Generated new TSK 0
16:11:12.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.437122 17781 catalog_manager.cc:1514] Initializing in-progress tserver states...
16:11:12.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.470347 17322 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu
16:11:12.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu
16:11:12.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-0/wal
16:11:12.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-0/data
16:11:12.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
16:11:12.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
16:11:12.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
16:11:12.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:11:12.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:11:12.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:11:12.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-0/logs
16:11:12.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-0/data/info.pb
16:11:12.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:11:12.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
16:11:12.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
16:11:12.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
16:11:12.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
16:11:12.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:11:12.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:11:12.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:11:12.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
16:11:12.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
16:11:12.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.234.129:0
16:11:12.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.16.234.129
16:11:12.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.234.129
16:11:12.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
16:11:12.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.16.234.190:33901,127.16.234.189:32987,127.16.234.188:38065
16:11:12.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.234.148:43519
16:11:12.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:11:12.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:11:12.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
16:11:12.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.490231 17594 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.234.188" port: 38065 } attempt: 7
16:11:12.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.537007 17594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5eaadf6df4764214a317ceb58603922b: Bootstrap starting.
16:11:12.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.562592 17594 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5eaadf6df4764214a317ceb58603922b: Neither blocks nor log segments found. Creating new log.
16:11:12.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.564916 17594 log.cc:824] T 00000000000000000000000000000000 P 5eaadf6df4764214a317ceb58603922b: Log is configured to *not* fsync() on all Append() calls
16:11:12.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.581797 17594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5eaadf6df4764214a317ceb58603922b: No bootstrap required, opened a new log
16:11:12.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.671000 17594 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5eaadf6df4764214a317ceb58603922b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ea4ce510c534787afa58a33455b3bff" member_type: VOTER last_known_addr { host: "127.16.234.190" port: 33901 } } peers { permanent_uuid: "5eaadf6df4764214a317ceb58603922b" member_type: VOTER last_known_addr { host: "127.16.234.189" port: 32987 } } peers { permanent_uuid: "cdd30a1bb4e2432e9b560c791116b727" member_type: VOTER last_known_addr { host: "127.16.234.188" port: 38065 } }
16:11:12.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.672201 17594 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5eaadf6df4764214a317ceb58603922b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
16:11:12.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.672720 17594 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5eaadf6df4764214a317ceb58603922b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5eaadf6df4764214a317ceb58603922b, State: Initialized, Role: FOLLOWER
16:11:12.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.673803 17594 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5eaadf6df4764214a317ceb58603922b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ea4ce510c534787afa58a33455b3bff" member_type: VOTER last_known_addr { host: "127.16.234.190" port: 33901 } } peers { permanent_uuid: "5eaadf6df4764214a317ceb58603922b" member_type: VOTER last_known_addr { host: "127.16.234.189" port: 32987 } } peers { permanent_uuid: "cdd30a1bb4e2432e9b560c791116b727" member_type: VOTER last_known_addr { host: "127.16.234.188" port: 38065 } }
16:11:12.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.677840 17795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5eaadf6df4764214a317ceb58603922b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ea4ce510c534787afa58a33455b3bff" member_type: VOTER last_known_addr { host: "127.16.234.190" port: 33901 } } peers { permanent_uuid: "5eaadf6df4764214a317ceb58603922b" member_type: VOTER last_known_addr { host: "127.16.234.189" port: 32987 } } peers { permanent_uuid: "cdd30a1bb4e2432e9b560c791116b727" member_type: VOTER last_known_addr { host: "127.16.234.188" port: 38065 } } }
16:11:12.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.679749 17795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5eaadf6df4764214a317ceb58603922b [sys.catalog]: This master's current role is: FOLLOWER
16:11:12.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.682819 17594 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5eaadf6df4764214a317ceb58603922b [sys.catalog]: configured and running, proceeding with master startup.
16:11:12.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.683007 17556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cdd30a1bb4e2432e9b560c791116b727" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5eaadf6df4764214a317ceb58603922b"
16:11:12.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.684468 17557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cdd30a1bb4e2432e9b560c791116b727" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5eaadf6df4764214a317ceb58603922b" is_pre_election: true
16:11:12.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.685649 17555 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5eaadf6df4764214a317ceb58603922b [term 0 FOLLOWER]: Advancing to term 1
16:11:12.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.696484 17555 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5eaadf6df4764214a317ceb58603922b [term 1 FOLLOWER]: Refusing update from remote peer cdd30a1bb4e2432e9b560c791116b727: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
16:11:12.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.698223 17778 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cdd30a1bb4e2432e9b560c791116b727 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eaadf6df4764214a317ceb58603922b" member_type: VOTER last_known_addr { host: "127.16.234.189" port: 32987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
16:11:12.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.758205 17795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5eaadf6df4764214a317ceb58603922b [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdd30a1bb4e2432e9b560c791116b727. Latest consensus state: current_term: 1 leader_uuid: "cdd30a1bb4e2432e9b560c791116b727" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ea4ce510c534787afa58a33455b3bff" member_type: VOTER last_known_addr { host: "127.16.234.190" port: 33901 } } peers { permanent_uuid: "5eaadf6df4764214a317ceb58603922b" member_type: VOTER last_known_addr { host: "127.16.234.189" port: 32987 } } peers { permanent_uuid: "cdd30a1bb4e2432e9b560c791116b727" member_type: VOTER last_known_addr { host: "127.16.234.188" port: 38065 } } }
16:11:12.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.758965 17795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5eaadf6df4764214a317ceb58603922b [sys.catalog]: This master's current role is: FOLLOWER
16:11:12.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.760289 17801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5eaadf6df4764214a317ceb58603922b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cdd30a1bb4e2432e9b560c791116b727" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ea4ce510c534787afa58a33455b3bff" member_type: VOTER last_known_addr { host: "127.16.234.190" port: 33901 } } peers { permanent_uuid: "5eaadf6df4764214a317ceb58603922b" member_type: VOTER last_known_addr { host: "127.16.234.189" port: 32987 } } peers { permanent_uuid: "cdd30a1bb4e2432e9b560c791116b727" member_type: VOTER last_known_addr { host: "127.16.234.188" port: 38065 } } }
16:11:12.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.761534 17801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5eaadf6df4764214a317ceb58603922b [sys.catalog]: This master's current role is: FOLLOWER
16:11:12.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.811074 17822 catalog_manager.cc:1259] Loaded cluster ID: b784a3075a744d64807f5591c6b86af3
16:11:12.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.811753 17822 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5eaadf6df4764214a317ceb58603922b: loading cluster ID for follower catalog manager: success
16:11:12.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.819803 17822 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5eaadf6df4764214a317ceb58603922b: acquiring CA information for follower catalog manager: success
16:11:12.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:12.825984 17822 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5eaadf6df4764214a317ceb58603922b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
16:11:13.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:12.999641 17790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
16:11:13.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:13.000665 17790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
16:11:13.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:13.001667 17790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
16:11:13.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:13.060282 17790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
16:11:13.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:13.061995 17790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.234.129
16:11:13.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:13.122725 17790 tablet_server_runner.cc:78] Tablet server non-default flags:
16:11:13.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:11:13.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.234.148:43519
16:11:13.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:11:13.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
16:11:13.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-0/data
16:11:13.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-0/wal
16:11:13.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:11:13.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:11:13.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.234.129:0
16:11:13.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
16:11:13.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:11:13.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:11:13.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-0/data/info.pb
16:11:13.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.234.129
16:11:13.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
16:11:13.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.16.234.190:33901,127.16.234.189:32987,127.16.234.188:38065
16:11:13.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
16:11:13.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:11:13.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
16:11:13.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
16:11:13.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:11:13.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.16.234.129
16:11:13.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-0/logs
16:11:13.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:11:13.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
16:11:13.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
16:11:13.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
16:11:13.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
16:11:13.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
16:11:13.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
16:11:13.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 15:12:52 UTC on ccycloud.quasar-tzrfpd.root.comops.site
16:11:13.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4580143
16:11:13.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
16:11:13.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:13.127000 17790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
16:11:13.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:13.129380 17790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
16:11:13.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:13.173169 17835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:11:13.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:13.183229 17776 catalog_manager.cc:1259] Loaded cluster ID: b784a3075a744d64807f5591c6b86af3
16:11:13.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:13.183856 17776 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2ea4ce510c534787afa58a33455b3bff: loading cluster ID for follower catalog manager: success
16:11:13.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:13.194633 17776 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2ea4ce510c534787afa58a33455b3bff: acquiring CA information for follower catalog manager: success
16:11:13.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:13.206660 17776 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2ea4ce510c534787afa58a33455b3bff: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
16:11:13.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:13.316413 17844 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdd30a1bb4e2432e9b560c791116b727 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cdd30a1bb4e2432e9b560c791116b727" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ea4ce510c534787afa58a33455b3bff" member_type: VOTER last_known_addr { host: "127.16.234.190" port: 33901 } } peers { permanent_uuid: "5eaadf6df4764214a317ceb58603922b" member_type: VOTER last_known_addr { host: "127.16.234.189" port: 32987 } } peers { permanent_uuid: "cdd30a1bb4e2432e9b560c791116b727" member_type: VOTER last_known_addr { host: "127.16.234.188" port: 38065 } } }
16:11:13.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:13.323666 17844 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdd30a1bb4e2432e9b560c791116b727 [sys.catalog]: This master's current role is: LEADER
16:11:14.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:13.198348 17790 server_base.cc:1034] running on GCE node
16:11:14.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:13.198602 17838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:11:14.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:13.195897 17834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:11:14.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:14.789578 17790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
16:11:14.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:14.795365 17790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
16:11:14.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:14.797209 17790 hybrid_clock.cc:648] HybridClock initialized: now 1786464674797172 us; error 50 us; skew 500 ppm
16:11:14.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:14.798450 17790 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
16:11:14.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:14.806772 17790 webserver.cc:460] Webserver started at http://127.16.234.129:43191/ using document root <none> and password file <none>
16:11:14.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:14.808015 17790 fs_manager.cc:359] Metadata directory not provided
16:11:14.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:14.808310 17790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
16:11:14.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:14.808887 17790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
16:11:14.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:14.815960 17790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-0/data/instance:
16:11:14.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "d9315253ea9e464c9416773b52ee00a6"
16:11:14.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 16:11:14 on dist-test-slave-zj8q"
16:11:14.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:14.817432 17790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-0/wal/instance:
16:11:14.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "d9315253ea9e464c9416773b52ee00a6"
16:11:14.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 16:11:14 on dist-test-slave-zj8q"
16:11:14.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:14.827132 17790 fs_manager.cc:691] Time spent creating directory manager: real 0.009s	user 0.011s	sys 0.001s
16:11:14.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:14.834739 17855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
16:11:14.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:14.836408 17790 fs_manager.cc:725] Time spent opening block manager: real 0.005s	user 0.004s	sys 0.000s
16:11:14.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:14.836795 17790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-0/data,/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-0/wal
16:11:14.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "d9315253ea9e464c9416773b52ee00a6"
16:11:14.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 16:11:14 on dist-test-slave-zj8q"
16:11:14.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:14.837227 17790 fs_report.cc:383] FS layout report
16:11:14.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
16:11:14.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-0/wal
16:11:14.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-0/wal
16:11:14.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-0/data/data
16:11:14.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
16:11:14.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
16:11:14.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
16:11:14.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
16:11:14.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
16:11:14.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
16:11:14.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
16:11:14.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
16:11:14.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
16:11:14.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
16:11:14.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
16:11:14.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:14.908442 17790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
16:11:14.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:14.910795 17790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
16:11:14.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:14.911679 17790 kserver.cc:163] Server-wide thread pool size limit: 3276
16:11:14.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:14.921561 17790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
16:11:14.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:14.934655 17790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
16:11:14.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:14.934934 17790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
16:11:14.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:14.935247 17790 ts_tablet_manager.cc:610] Registered 0 tablets
16:11:14.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:14.935465 17790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
16:11:15.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:15.124425 17790 rpc_server.cc:307] RPC server started. Bound to: 127.16.234.129:42901
16:11:15.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:15.124481 18050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.234.129:42901 every 8 connection(s)
16:11:15.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:15.128086 17790 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-0/data/info.pb
16:11:15.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:15.130931 17322 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu as pid 17790
16:11:15.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:15.131390 17322 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-0/wal/instance
16:11:15.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:15.153040 17322 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu
16:11:15.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu
16:11:15.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/wal
16:11:15.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/data
16:11:15.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
16:11:15.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
16:11:15.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
16:11:15.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:11:15.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:11:15.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:11:15.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/logs
16:11:15.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/data/info.pb
16:11:15.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:11:15.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
16:11:15.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
16:11:15.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
16:11:15.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
16:11:15.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:11:15.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:11:15.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:11:15.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
16:11:15.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
16:11:15.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.234.130:0
16:11:15.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.16.234.130
16:11:15.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.234.130
16:11:15.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
16:11:15.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.16.234.190:33901,127.16.234.189:32987,127.16.234.188:38065
16:11:15.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.234.148:43519
16:11:15.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:11:15.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:11:15.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
16:11:15.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:15.206681 18051 heartbeater.cc:347] Connected to a master server at 127.16.234.188:38065
16:11:15.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:15.207201 18051 heartbeater.cc:464] Registering TS with master...
16:11:15.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:15.208402 18051 heartbeater.cc:511] Master 127.16.234.188:38065 requested a full tablet report, sending...
16:11:15.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:15.210261 18057 heartbeater.cc:347] Connected to a master server at 127.16.234.190:33901
16:11:15.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:15.210597 18057 heartbeater.cc:464] Registering TS with master...
16:11:15.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:15.211398 18057 heartbeater.cc:511] Master 127.16.234.190:33901 requested a full tablet report, sending...
16:11:15.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:15.211535 17662 ts_manager.cc:194] Registered new tserver with Master: d9315253ea9e464c9416773b52ee00a6 (127.16.234.129:42901)
16:11:15.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:15.214403 17662 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.234.129:54619
16:11:15.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:15.214516 17402 ts_manager.cc:194] Registered new tserver with Master: d9315253ea9e464c9416773b52ee00a6 (127.16.234.129:42901)
16:11:15.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:15.217725 18053 heartbeater.cc:347] Connected to a master server at 127.16.234.189:32987
16:11:15.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:15.218050 18053 heartbeater.cc:464] Registering TS with master...
16:11:15.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:15.218742 18053 heartbeater.cc:511] Master 127.16.234.189:32987 requested a full tablet report, sending...
16:11:15.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:15.221830 17537 ts_manager.cc:194] Registered new tserver with Master: d9315253ea9e464c9416773b52ee00a6 (127.16.234.129:42901)
16:11:15.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:15.500885 18056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
16:11:15.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:15.501412 18056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
16:11:15.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:15.501971 18056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
16:11:15.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:15.533653 18056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
16:11:15.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:15.534721 18056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.234.130
16:11:15.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:15.568889 18056 tablet_server_runner.cc:78] Tablet server non-default flags:
16:11:15.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:11:15.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.234.148:43519
16:11:15.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:11:15.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
16:11:15.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/data
16:11:15.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/wal
16:11:15.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:11:15.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:11:15.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.234.130:0
16:11:15.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
16:11:15.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:11:15.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:11:15.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/data/info.pb
16:11:15.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.234.130
16:11:15.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
16:11:15.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.16.234.190:33901,127.16.234.189:32987,127.16.234.188:38065
16:11:15.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
16:11:15.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:11:15.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
16:11:15.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
16:11:15.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:11:15.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.16.234.130
16:11:15.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/logs
16:11:15.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:11:15.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
16:11:15.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
16:11:15.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
16:11:15.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
16:11:15.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
16:11:15.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
16:11:15.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 15:12:52 UTC on ccycloud.quasar-tzrfpd.root.comops.site
16:11:15.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4580143
16:11:15.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
16:11:15.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:15.570415 18056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
16:11:15.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:15.572388 18056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
16:11:15.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:15.594002 18081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:11:16.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:16.219089 18051 heartbeater.cc:503] Master 127.16.234.188:38065 was elected leader, sending a full tablet report...
16:11:16.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:15.594013 18075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:11:16.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:15.594297 18076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:11:16.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:15.596225 18056 server_base.cc:1034] running on GCE node
16:11:16.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:16.689085 18056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
16:11:16.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:16.691721 18056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
16:11:16.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:16.693149 18056 hybrid_clock.cc:648] HybridClock initialized: now 1786464676693119 us; error 44 us; skew 500 ppm
16:11:16.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:16.693902 18056 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
16:11:16.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:16.700949 18056 webserver.cc:460] Webserver started at http://127.16.234.130:37863/ using document root <none> and password file <none>
16:11:16.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:16.701903 18056 fs_manager.cc:359] Metadata directory not provided
16:11:16.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:16.702122 18056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
16:11:16.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:16.702673 18056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
16:11:16.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:16.707531 18056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/data/instance:
16:11:16.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "1bce670edc6c40b2bb1584ec3b364e0f"
16:11:16.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 16:11:16 on dist-test-slave-zj8q"
16:11:16.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:16.708643 18056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/wal/instance:
16:11:16.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "1bce670edc6c40b2bb1584ec3b364e0f"
16:11:16.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 16:11:16 on dist-test-slave-zj8q"
16:11:16.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:16.715499 18056 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.008s	sys 0.000s
16:11:16.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:16.721771 18093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
16:11:16.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:16.722821 18056 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
16:11:16.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:16.723104 18056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/data,/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/wal
16:11:16.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "1bce670edc6c40b2bb1584ec3b364e0f"
16:11:16.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 16:11:16 on dist-test-slave-zj8q"
16:11:16.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:16.723649 18056 fs_report.cc:383] FS layout report
16:11:16.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
16:11:16.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/wal
16:11:16.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/wal
16:11:16.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/data/data
16:11:16.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
16:11:16.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
16:11:16.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
16:11:16.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
16:11:16.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
16:11:16.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
16:11:16.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
16:11:16.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
16:11:16.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
16:11:16.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
16:11:16.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
16:11:16.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:16.757573 18056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
16:11:16.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:16.759097 18056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
16:11:16.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:16.759598 18056 kserver.cc:163] Server-wide thread pool size limit: 3276
16:11:16.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:16.762315 18056 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
16:11:16.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:16.766583 18056 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
16:11:16.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:16.766786 18056 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
16:11:16.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:16.767096 18056 ts_tablet_manager.cc:610] Registered 0 tablets
16:11:16.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:16.767285 18056 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
16:11:16.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:16.912469 18056 rpc_server.cc:307] RPC server started. Bound to: 127.16.234.130:43937
16:11:16.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:16.912571 18290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.234.130:43937 every 8 connection(s)
16:11:16.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:16.915077 18056 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/data/info.pb
16:11:16.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:16.923710 17322 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu as pid 18056
16:11:16.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:16.924415 17322 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/wal/instance
16:11:16.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:16.943445 17322 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu
16:11:16.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu
16:11:16.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-2/wal
16:11:16.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-2/data
16:11:16.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
16:11:16.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
16:11:16.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
16:11:16.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:11:16.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:11:16.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:11:16.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-2/logs
16:11:16.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-2/data/info.pb
16:11:16.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:11:16.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
16:11:16.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
16:11:16.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
16:11:16.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
16:11:16.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:11:16.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:11:16.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:11:16.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
16:11:16.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
16:11:16.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.234.131:0
16:11:16.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.16.234.131
16:11:16.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.234.131
16:11:16.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
16:11:16.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.16.234.190:33901,127.16.234.189:32987,127.16.234.188:38065
16:11:16.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.234.148:43519
16:11:16.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:11:16.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:11:16.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
16:11:16.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:16.966931 18303 heartbeater.cc:347] Connected to a master server at 127.16.234.190:33901
16:11:16.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:16.967484 18303 heartbeater.cc:464] Registering TS with master...
16:11:16.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:16.968757 18303 heartbeater.cc:511] Master 127.16.234.190:33901 requested a full tablet report, sending...
16:11:16.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:16.971415 18291 heartbeater.cc:347] Connected to a master server at 127.16.234.188:38065
16:11:16.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:16.971755 18291 heartbeater.cc:464] Registering TS with master...
16:11:16.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:16.972496 17402 ts_manager.cc:194] Registered new tserver with Master: 1bce670edc6c40b2bb1584ec3b364e0f (127.16.234.130:43937)
16:11:16.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:16.972585 18291 heartbeater.cc:511] Master 127.16.234.188:38065 requested a full tablet report, sending...
16:11:16.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:16.974625 17662 ts_manager.cc:194] Registered new tserver with Master: 1bce670edc6c40b2bb1584ec3b364e0f (127.16.234.130:43937)
16:11:16.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:16.976631 17662 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.234.130:57113
16:11:16.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:16.981869 18299 heartbeater.cc:347] Connected to a master server at 127.16.234.189:32987
16:11:16.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:16.982121 18299 heartbeater.cc:464] Registering TS with master...
16:11:16.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:16.982713 18299 heartbeater.cc:511] Master 127.16.234.189:32987 requested a full tablet report, sending...
16:11:16.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:16.984867 17537 ts_manager.cc:194] Registered new tserver with Master: 1bce670edc6c40b2bb1584ec3b364e0f (127.16.234.130:43937)
16:11:17.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:17.272027 18318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
16:11:17.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:17.272565 18318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
16:11:17.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:17.273041 18318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
16:11:17.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:17.304124 18318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
16:11:17.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:17.304976 18318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.234.131
16:11:17.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:17.339074 18318 tablet_server_runner.cc:78] Tablet server non-default flags:
16:11:17.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:11:17.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.234.148:43519
16:11:17.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:11:17.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
16:11:17.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-2/data
16:11:17.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-2/wal
16:11:17.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:11:17.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:11:17.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.234.131:0
16:11:17.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
16:11:17.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:11:17.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:11:17.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-2/data/info.pb
16:11:17.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.234.131
16:11:17.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
16:11:17.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.16.234.190:33901,127.16.234.189:32987,127.16.234.188:38065
16:11:17.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
16:11:17.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:11:17.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
16:11:17.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
16:11:17.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:11:17.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.16.234.131
16:11:17.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-2/logs
16:11:17.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:11:17.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
16:11:17.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
16:11:17.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
16:11:17.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
16:11:17.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
16:11:17.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
16:11:17.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 15:12:52 UTC on ccycloud.quasar-tzrfpd.root.comops.site
16:11:17.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4580143
16:11:17.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
16:11:17.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:17.340226 18318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
16:11:17.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:17.341900 18318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
16:11:17.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:17.361423 18328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:11:17.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:17.980515 18291 heartbeater.cc:503] Master 127.16.234.188:38065 was elected leader, sending a full tablet report...
16:11:18.446 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:17.361495 18329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:11:18.446 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:17.364127 18318 server_base.cc:1034] running on GCE node
16:11:18.446 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:17.362296 18332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:11:18.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:18.450026 18318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
16:11:18.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:18.452332 18318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
16:11:18.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:18.453676 18318 hybrid_clock.cc:648] HybridClock initialized: now 1786464678453646 us; error 49 us; skew 500 ppm
16:11:18.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:18.454607 18318 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
16:11:18.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:18.460749 18318 webserver.cc:460] Webserver started at http://127.16.234.131:35277/ using document root <none> and password file <none>
16:11:18.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:18.461637 18318 fs_manager.cc:359] Metadata directory not provided
16:11:18.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:18.461822 18318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
16:11:18.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:18.462261 18318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
16:11:18.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:18.467113 18318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-2/data/instance:
16:11:18.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "3d392ba816c24499a0f7fecd95f88c15"
16:11:18.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 16:11:18 on dist-test-slave-zj8q"
16:11:18.468 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:18.468144 18318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-2/wal/instance:
16:11:18.468 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "3d392ba816c24499a0f7fecd95f88c15"
16:11:18.468 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 16:11:18 on dist-test-slave-zj8q"
16:11:18.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:18.475176 18318 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.007s	sys 0.001s
16:11:18.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:18.480551 18344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
16:11:18.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:18.481560 18318 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.005s	sys 0.000s
16:11:18.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:18.481899 18318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-2/data,/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-2/wal
16:11:18.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "3d392ba816c24499a0f7fecd95f88c15"
16:11:18.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 16:11:18 on dist-test-slave-zj8q"
16:11:18.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:18.482286 18318 fs_report.cc:383] FS layout report
16:11:18.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
16:11:18.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-2/wal
16:11:18.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-2/wal
16:11:18.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-2/data/data
16:11:18.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
16:11:18.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
16:11:18.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
16:11:18.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
16:11:18.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
16:11:18.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
16:11:18.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
16:11:18.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
16:11:18.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
16:11:18.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
16:11:18.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
16:11:18.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:18.521920 18318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
16:11:18.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:18.523505 18318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
16:11:18.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:18.524020 18318 kserver.cc:163] Server-wide thread pool size limit: 3276
16:11:18.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:18.526804 18318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
16:11:18.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:18.530864 18318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
16:11:18.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:18.531065 18318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
16:11:18.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:18.531379 18318 ts_tablet_manager.cc:610] Registered 0 tablets
16:11:18.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:18.531545 18318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
16:11:18.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:18.815416 18318 rpc_server.cc:307] RPC server started. Bound to: 127.16.234.131:42843
16:11:18.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:18.815526 18553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.234.131:42843 every 8 connection(s)
16:11:18.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:18.818109 18318 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-2/data/info.pb
16:11:18.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:18.821748 17322 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu as pid 18318
16:11:18.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:18.822548 17322 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-2/wal/instance
16:11:18.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:18.911432 18557 heartbeater.cc:347] Connected to a master server at 127.16.234.188:38065
16:11:18.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:18.912176 18557 heartbeater.cc:464] Registering TS with master...
16:11:18.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:18.914459 18557 heartbeater.cc:511] Master 127.16.234.188:38065 requested a full tablet report, sending...
16:11:18.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:18.917671 18558 heartbeater.cc:347] Connected to a master server at 127.16.234.189:32987
16:11:18.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:18.918200 18558 heartbeater.cc:464] Registering TS with master...
16:11:18.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:18.919423 17662 ts_manager.cc:194] Registered new tserver with Master: 3d392ba816c24499a0f7fecd95f88c15 (127.16.234.131:42843)
16:11:18.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:18.919982 18558 heartbeater.cc:511] Master 127.16.234.189:32987 requested a full tablet report, sending...
16:11:18.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:18.922806 17662 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.234.131:36315
16:11:18.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:18.924124 18563 heartbeater.cc:347] Connected to a master server at 127.16.234.190:33901
16:11:18.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:18.924676 18563 heartbeater.cc:464] Registering TS with master...
16:11:18.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:18.926553 18563 heartbeater.cc:511] Master 127.16.234.190:33901 requested a full tablet report, sending...
16:11:18.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:18.933084 17402 ts_manager.cc:194] Registered new tserver with Master: 3d392ba816c24499a0f7fecd95f88c15 (127.16.234.131:42843)
16:11:18.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:18.935534 17537 ts_manager.cc:194] Registered new tserver with Master: 3d392ba816c24499a0f7fecd95f88c15 (127.16.234.131:42843)
16:11:18.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:18.954849 17322 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
16:11:18.965 [DEBUG - main] (MiniKuduCluster.java:195) Response: 
16:11:18.967 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_masters {
}

16:11:18.976 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_masters {
  masters {
    id {
      type: MASTER
      index: 0
    }
    bound_rpc_address {
      host: "127.16.234.190"
      port: 33901
    }
    bound_http_address {
      host: "127.16.234.190"
      port: 40811
    }
  }
  masters {
    id {
      type: MASTER
      index: 1
    }
    bound_rpc_address {
      host: "127.16.234.189"
      port: 32987
    }
    bound_http_address {
      host: "127.16.234.189"
      port: 33193
    }
  }
  masters {
    id {
      type: MASTER
      index: 2
    }
    bound_rpc_address {
      host: "127.16.234.188"
      port: 38065
    }
    bound_http_address {
      host: "127.16.234.188"
      port: 38981
    }
  }
}

16:11:18.983 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_tservers {
}

16:11:18.986 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_tservers {
  tservers {
    id {
      type: TSERVER
      index: 0
    }
    bound_rpc_address {
      host: "127.16.234.129"
      port: 42901
    }
    bound_http_address {
      host: "127.16.234.129"
      port: 43191
    }
  }
  tservers {
    id {
      type: TSERVER
      index: 1
    }
    bound_rpc_address {
      host: "127.16.234.130"
      port: 43937
    }
    bound_http_address {
      host: "127.16.234.130"
      port: 37863
    }
  }
  tservers {
    id {
      type: TSERVER
      index: 2
    }
    bound_rpc_address {
      host: "127.16.234.131"
      port: 42843
    }
    bound_http_address {
      host: "127.16.234.131"
      port: 35277
    }
  }
}

16:11:18.986 [INFO - main] (KuduTestHarness.java:157) Creating a new Kudu client...
16:11:19.167 [DEBUG - main] (SecurityUtil.java:122) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 
16:11:19.355 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.16.234.189:32987(127.16.234.189:32987)] connecting to peer
16:11:19.408 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.16.234.188:38065(127.16.234.188:38065)] connecting to peer
16:11:19.410 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.16.234.190:33901(127.16.234.190:33901)] connecting to peer
16:11:19.447 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.16.234.188:38065(127.16.234.188:38065)] Successfully connected to peer
16:11:19.448 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.16.234.189:32987(127.16.234.189:32987)] Successfully connected to peer
16:11:19.519 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.16.234.190:33901(127.16.234.190:33901)] Successfully connected to peer
16:11:19.554 [DEBUG - kudu-nio-1] (Negotiator.java:529) SASL mechanism PLAIN chosen for peer 127.16.234.188
16:11:19.555 [DEBUG - kudu-nio-0] (Negotiator.java:529) SASL mechanism PLAIN chosen for peer 127.16.234.189
16:11:19.679 [DEBUG - kudu-nio-0] (Negotiator.java:529) SASL mechanism PLAIN chosen for peer 127.16.234.190
16:11:19.811 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x94179357, L:/127.0.0.1:34742 - R:/127.16.234.188:38065] using SASL/PLAIN
16:11:19.811 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0x8cb65958, L:/127.0.0.1:37130 - R:/127.16.234.189:32987] using SASL/PLAIN
16:11:19.827 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0x56c987ef, L:/127.0.0.1:58660 - R:/127.16.234.190:33901] using SASL/PLAIN
16:11:19.884 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet Kudu Master for table 'Kudu Master' with partition [<start>, <end>)
16:11:19.886 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=CreateTable, tablet=null, attempt=1, TimeoutTracker(timeout=50000, elapsed=570), Traces: [0ms] refreshing cache from master, [27ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.234.189:32987, [88ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.234.188:38065, [90ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.234.190:33901, [517ms] Sub RPC ConnectToMaster: received response from server master-127.16.234.189:32987: OK, [518ms] Sub RPC ConnectToMaster: received response from server master-127.16.234.188:38065: OK, [533ms] Sub RPC ConnectToMaster: received response from server master-127.16.234.190:33901: OK, deferred=null) after lookup
16:11:19.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:19.931337 18557 heartbeater.cc:503] Master 127.16.234.188:38065 was elected leader, sending a full tablet report...
16:11:19.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:19.945179 17662 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
16:11:19.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) name: "org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304"
16:11:19.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) schema {
16:11:19.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
16:11:19.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "key"
16:11:19.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
16:11:19.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: true
16:11:19.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
16:11:19.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
16:11:19.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
16:11:19.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
16:11:19.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
16:11:19.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
16:11:19.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column1_i"
16:11:19.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
16:11:19.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
16:11:19.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
16:11:19.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
16:11:19.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
16:11:19.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
16:11:19.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
16:11:19.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
16:11:19.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column2_i"
16:11:19.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
16:11:19.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
16:11:19.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
16:11:19.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
16:11:19.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
16:11:19.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
16:11:19.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
16:11:19.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
16:11:19.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column3_s"
16:11:19.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: STRING
16:11:19.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
16:11:19.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: true
16:11:19.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     encoding: DICT_ENCODING
16:11:19.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     compression: LZ4
16:11:19.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 4096
16:11:19.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
16:11:19.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
16:11:19.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
16:11:19.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
16:11:19.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column4_b"
16:11:19.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: BOOL
16:11:19.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
16:11:19.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
16:11:19.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
16:11:19.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
16:11:19.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
16:11:19.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
16:11:19.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) }
16:11:19.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) partition_schema {
16:11:19.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   range_schema {
16:11:19.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     columns {
16:11:19.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)       name: "key"
16:11:19.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     }
16:11:19.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
16:11:19.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) }
16:11:19.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:19.952258 17662 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
16:11:20.056 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:894) Opened table 4788c1a799fc46278fed8e3889d5551a
16:11:20.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.122316 18432 tablet_service.cc:1467] Processing CreateTablet for tablet 7d3c128d101444a59e009013a5c634dc (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304 [id=4788c1a799fc46278fed8e3889d5551a]), partition=RANGE (key) PARTITION UNBOUNDED
16:11:20.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.125720 18432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d3c128d101444a59e009013a5c634dc. 1 dirs total, 0 dirs full, 0 dirs failed
16:11:20.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.136149 17934 tablet_service.cc:1467] Processing CreateTablet for tablet 7d3c128d101444a59e009013a5c634dc (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304 [id=4788c1a799fc46278fed8e3889d5551a]), partition=RANGE (key) PARTITION UNBOUNDED
16:11:20.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.139075 17934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d3c128d101444a59e009013a5c634dc. 1 dirs total, 0 dirs full, 0 dirs failed
16:11:20.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.151832 18175 tablet_service.cc:1467] Processing CreateTablet for tablet 7d3c128d101444a59e009013a5c634dc (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304 [id=4788c1a799fc46278fed8e3889d5551a]), partition=RANGE (key) PARTITION UNBOUNDED
16:11:20.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.155376 18175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d3c128d101444a59e009013a5c634dc. 1 dirs total, 0 dirs full, 0 dirs failed
16:11:20.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.192927 18625 tablet_bootstrap.cc:492] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15: Bootstrap starting.
16:11:20.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.195590 18626 tablet_bootstrap.cc:492] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6: Bootstrap starting.
16:11:20.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:20.205228 18282 debug-util.cc:398] Leaking SignalData structure 0x7b08000bcca0 after lost signal to thread 18068
16:11:20.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.211124 18625 tablet_bootstrap.cc:654] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15: Neither blocks nor log segments found. Creating new log.
16:11:20.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.218567 18625 log.cc:824] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15: Log is configured to *not* fsync() on all Append() calls
16:11:20.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.229552 18626 tablet_bootstrap.cc:654] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6: Neither blocks nor log segments found. Creating new log.
16:11:20.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.231420 18631 tablet_bootstrap.cc:492] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f: Bootstrap starting.
16:11:20.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.238569 18626 log.cc:824] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6: Log is configured to *not* fsync() on all Append() calls
16:11:20.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.250229 18625 tablet_bootstrap.cc:492] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15: No bootstrap required, opened a new log
16:11:20.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.250949 18625 ts_tablet_manager.cc:1397] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15: Time spent bootstrapping tablet: real 0.059s	user 0.008s	sys 0.027s
16:11:20.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.265285 18631 tablet_bootstrap.cc:654] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f: Neither blocks nor log segments found. Creating new log.
16:11:20.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.267709 18626 tablet_bootstrap.cc:492] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6: No bootstrap required, opened a new log
16:11:20.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.268406 18626 ts_tablet_manager.cc:1397] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6: Time spent bootstrapping tablet: real 0.074s	user 0.006s	sys 0.030s
16:11:20.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.272104 18631 log.cc:824] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f: Log is configured to *not* fsync() on all Append() calls
16:11:20.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.289577 18631 tablet_bootstrap.cc:492] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f: No bootstrap required, opened a new log
16:11:20.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.290391 18631 ts_tablet_manager.cc:1397] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f: Time spent bootstrapping tablet: real 0.065s	user 0.022s	sys 0.019s
16:11:20.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.319545 18625 raft_consensus.cc:348] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:11:20.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.321053 18625 raft_consensus.cc:374] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
16:11:20.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.321626 18625 raft_consensus.cc:729] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d392ba816c24499a0f7fecd95f88c15, State: Initialized, Role: FOLLOWER
16:11:20.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.324937 18625 consensus_queue.cc:260] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:11:20.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.337212 18625 ts_tablet_manager.cc:1428] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15: Time spent starting tablet: real 0.086s	user 0.052s	sys 0.022s
16:11:20.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.346189 18626 raft_consensus.cc:348] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:11:20.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.347640 18626 raft_consensus.cc:374] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
16:11:20.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.348043 18626 raft_consensus.cc:729] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9315253ea9e464c9416773b52ee00a6, State: Initialized, Role: FOLLOWER
16:11:20.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.351271 18626 consensus_queue.cc:260] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:11:20.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.353048 18631 raft_consensus.cc:348] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:11:20.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.354437 18631 raft_consensus.cc:374] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
16:11:20.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.354908 18631 raft_consensus.cc:729] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bce670edc6c40b2bb1584ec3b364e0f, State: Initialized, Role: FOLLOWER
16:11:20.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.359757 18631 consensus_queue.cc:260] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:11:20.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:20.370924 18572 tablet.cc:2369] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15: Can't schedule compaction. Clean time has not been advanced past its initial value.
16:11:20.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.374588 18631 ts_tablet_manager.cc:1428] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f: Time spent starting tablet: real 0.084s	user 0.027s	sys 0.051s
16:11:20.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.394279 18626 ts_tablet_manager.cc:1428] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6: Time spent starting tablet: real 0.125s	user 0.051s	sys 0.028s
16:11:20.435 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:20.434542 18304 tablet.cc:2369] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f: Can't schedule compaction. Clean time has not been advanced past its initial value.
16:11:20.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:20.444778 18067 tablet.cc:2369] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
16:11:20.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.497282 18644 raft_consensus.cc:482] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
16:11:20.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.497926 18644 raft_consensus.cc:504] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:11:20.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.500777 18644 leader_election.cc:288] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d392ba816c24499a0f7fecd95f88c15 (127.16.234.131:42843), d9315253ea9e464c9416773b52ee00a6 (127.16.234.129:42901)
16:11:20.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.518043 18472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d3c128d101444a59e009013a5c634dc" candidate_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d392ba816c24499a0f7fecd95f88c15" is_pre_election: true
16:11:20.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.518998 18472 raft_consensus.cc:2454] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bce670edc6c40b2bb1584ec3b364e0f in term 0.
16:11:20.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.520833 18096 leader_election.cc:302] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bce670edc6c40b2bb1584ec3b364e0f, 3d392ba816c24499a0f7fecd95f88c15; no voters: 
16:11:20.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.521791 18644 raft_consensus.cc:2781] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [term 0 FOLLOWER]: Leader pre-election won for term 1
16:11:20.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.522106 18644 raft_consensus.cc:482] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
16:11:20.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.522413 18644 raft_consensus.cc:3037] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [term 0 FOLLOWER]: Advancing to term 1
16:11:20.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.525089 17972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d3c128d101444a59e009013a5c634dc" candidate_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9315253ea9e464c9416773b52ee00a6" is_pre_election: true
16:11:20.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.526002 17972 raft_consensus.cc:2454] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bce670edc6c40b2bb1584ec3b364e0f in term 0.
16:11:20.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.529165 18644 raft_consensus.cc:504] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:11:20.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.532174 18472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d3c128d101444a59e009013a5c634dc" candidate_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d392ba816c24499a0f7fecd95f88c15"
16:11:20.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.532732 18472 raft_consensus.cc:3037] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 0 FOLLOWER]: Advancing to term 1
16:11:20.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.532732 17972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d3c128d101444a59e009013a5c634dc" candidate_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9315253ea9e464c9416773b52ee00a6"
16:11:20.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.533279 17972 raft_consensus.cc:3037] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 0 FOLLOWER]: Advancing to term 1
16:11:20.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.539390 18472 raft_consensus.cc:2454] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bce670edc6c40b2bb1584ec3b364e0f in term 1.
16:11:20.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.531785 18644 leader_election.cc:288] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [CANDIDATE]: Term 1 election: Requested vote from peers 3d392ba816c24499a0f7fecd95f88c15 (127.16.234.131:42843), d9315253ea9e464c9416773b52ee00a6 (127.16.234.129:42901)
16:11:20.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.540591 17972 raft_consensus.cc:2454] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bce670edc6c40b2bb1584ec3b364e0f in term 1.
16:11:20.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.541145 18096 leader_election.cc:302] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bce670edc6c40b2bb1584ec3b364e0f, 3d392ba816c24499a0f7fecd95f88c15; no voters: 
16:11:20.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.542564 18644 raft_consensus.cc:2781] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [term 1 FOLLOWER]: Leader election won for term 1
16:11:20.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.547241 18644 raft_consensus.cc:686] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [term 1 LEADER]: Becoming Leader. State: Replica: 1bce670edc6c40b2bb1584ec3b364e0f, State: Running, Role: LEADER
16:11:20.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.548313 18644 consensus_queue.cc:237] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:11:20.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.577842 17662 catalog_manager.cc:5495] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f reported cstate change: term changed from 0 to 1, leader changed from <none> to 1bce670edc6c40b2bb1584ec3b364e0f (127.16.234.130). New cstate: current_term: 1 leader_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } health_report { overall_health: HEALTHY } } }
16:11:20.808 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:894) Opened table 4788c1a799fc46278fed8e3889d5551a
16:11:20.849 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:11:20.860 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299989, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)[L],3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843),d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)]}]
16:11:20.864 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=null, attempt=1, sequence_id=-1, TimeoutTracker(timeout=30000, elapsed=39), Traces: [0ms] refreshing cache from master, [0ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [15ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=null) after lookup
16:11:20.880 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] connecting to peer
16:11:20.883 [DEBUG - kudu-nio-1] (Connection.java:757) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] Successfully connected to peer
16:11:20.966 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x660b7fc5, L:/127.0.0.1:55612 - R:/127.16.234.130:43937] using TOKEN/null
16:11:20.984 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.983570 17972 raft_consensus.cc:1261] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 1 FOLLOWER]: Refusing update from remote peer 1bce670edc6c40b2bb1584ec3b364e0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
16:11:20.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.984676 18472 raft_consensus.cc:1261] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 1 FOLLOWER]: Refusing update from remote peer 1bce670edc6c40b2bb1584ec3b364e0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
16:11:20.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.985671 18644 consensus_queue.cc:1035] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
16:11:20.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:20.986862 18651 consensus_queue.cc:1035] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
16:11:21.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:21.046895 18673 mvcc.cc:205] Tried to move back new op lower bound from 7317359333291110400 to 7317359331541639168. Current Snapshot: MvccSnapshot[applied={T|T < 7317359333291110400 or (T in {7317359333291110400})}]
16:11:21.200 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 3 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304, tablet=7d3c128d101444a59e009013a5c634dc, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
16:11:21.203 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.16.234.130:43937
16:11:21.205 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

16:11:21.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:21.207713 17322 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu with pid 18056
16:11:21.273 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
16:11:21.276 [DEBUG - kudu-nio-1] (Connection.java:239) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] handling channelInactive
16:11:21.276 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] cleaning up while in state READY due to: connection closed
16:11:21.277 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937) for tablet 7d3c128d101444a59e009013a5c634dc: connection closed
16:11:21.301 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:11:21.302 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)[L],3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843),d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)]}]
16:11:21.303 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=2, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=29), Traces: [0ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [4ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: connection closed, [4ms] delaying RPC due to: Network error: connection closed, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [28ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@274916208(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@686091c0 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@31e9c516 -> wakeup thread Time-limited test)) after lookup
16:11:21.304 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] connecting to peer
16:11:21.306 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:11:21.306 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:11:21.306 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:11:21.321 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:11:21.321 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)[L],3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843),d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)]}]
16:11:21.321 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=4, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=48), Traces: [0ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [4ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: connection closed, [4ms] delaying RPC due to: Network error: connection closed, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [28ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [30ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [33ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [34ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [43ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [47ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@274916208(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@686091c0 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1775610208 after retry RPC@376111161), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@31e9c516 -> wakeup thread Time-limited test -> (continuation of Deferred@1775610208 after retry RPC@376111161))) after lookup
16:11:21.322 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] connecting to peer
16:11:21.323 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:11:21.323 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:11:21.323 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:11:21.342 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:11:21.342 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)[L],3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843),d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)]}]
16:11:21.343 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=6, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=69), Traces: [0ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [4ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: connection closed, [4ms] delaying RPC due to: Network error: connection closed, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [28ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [30ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [33ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [34ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [43ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [47ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [49ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [50ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [50ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [63ms] refreshing cache from master, [63ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [68ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@274916208(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@686091c0 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1775610208 after retry RPC@376111161) -> (continuation of Deferred@1995913747 after retry RPC@332941615), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@31e9c516 -> wakeup thread Time-limited test -> (continuation of Deferred@1775610208 after retry RPC@376111161) -> (continuation of Deferred@1995913747 after retry RPC@332941615))) after lookup
16:11:21.344 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] connecting to peer
16:11:21.346 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:11:21.346 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:11:21.346 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:11:21.361 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:11:21.362 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)[L],3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843),d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)]}]
16:11:21.362 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=8, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=89), Traces: [0ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [4ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: connection closed, [4ms] delaying RPC due to: Network error: connection closed, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [28ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [30ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [33ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [34ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [43ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [47ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [49ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [50ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [50ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [63ms] refreshing cache from master, [63ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [68ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [71ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [73ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [74ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [84ms] refreshing cache from master, [84ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [87ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@274916208(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@686091c0 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1775610208 after retry RPC@376111161) -> (continuation of Deferred@1995913747 after retry RPC@332941615) -> (continuation of Deferred@936185223 after retry RPC@1720564576), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@31e9c516 -> wakeup thread Time-limited test -> (continuation of Deferred@1775610208 after retry RPC@376111161) -> (continuation of Deferred@1995913747 after retry RPC@332941615) -> (continuation of Deferred@936185223 after retry RPC@1720564576))) after lookup
16:11:21.363 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] connecting to peer
16:11:21.364 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:11:21.364 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:11:21.365 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:11:21.562 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:11:21.563 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)[L],3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843),d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)]}]
16:11:21.564 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=10, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=290), Traces: [0ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [4ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: connection closed, [4ms] delaying RPC due to: Network error: connection closed, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [28ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [30ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [33ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [34ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [43ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [47ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [49ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [50ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [50ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [63ms] refreshing cache from master, [63ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [68ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [71ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [73ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [74ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [84ms] refreshing cache from master, [84ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [87ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [90ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [92ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [92ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [284ms] refreshing cache from master, [284ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [288ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@274916208(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@686091c0 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1775610208 after retry RPC@376111161) -> (continuation of Deferred@1995913747 after retry RPC@332941615) -> (continuation of Deferred@936185223 after retry RPC@1720564576) -> (continuation of Deferred@325465858 after retry RPC@719758382), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@31e9c516 -> wakeup thread Time-limited test -> (continuation of Deferred@1775610208 after retry RPC@376111161) -> (continuation of Deferred@1995913747 after retry RPC@332941615) -> (continuation of Deferred@936185223 after retry RPC@1720564576) -> (continuation of Deferred@325465858 after retry RPC@719758382))) after lookup
16:11:21.565 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] connecting to peer
16:11:21.567 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:11:21.567 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:11:21.567 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:11:21.921 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:11:21.921 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)[L],3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843),d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)]}]
16:11:21.922 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=12, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=648), Traces: [0ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [4ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: connection closed, [4ms] delaying RPC due to: Network error: connection closed, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [28ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [30ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [33ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [34ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [43ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [47ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [49ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [50ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [50ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [63ms] refreshing cache from master, [63ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [68ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [71ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [73ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [74ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [84ms] refreshing cache from master, [84ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [87ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [90ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [92ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [92ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [284ms] refreshing cache from master, [284ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [288ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [291ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [294ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [294ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [643ms] refreshing cache from master, [643ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [647ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@274916208(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@686091c0 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1775610208 after retry RPC@376111161) -> (continuation of Deferred@1995913747 after retry RPC@332941615) -> (continuation of Deferred@936185223 after retry RPC@1720564576) -> (continuation of Deferred@325465858 after retry RPC@719758382) -> (continuation of Deferred@1820092240 after retry RPC@1931766053), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@31e9c516 -> wakeup thread Time-limited test -> (continuation of Deferred@1775610208 after retry RPC@376111161) -> (continuation of Deferred@1995913747 after retry RPC@332941615) -> (continuation of Deferred@936185223 after retry RPC@1720564576) -> (continuation of Deferred@325465858 after retry RPC@719758382) -> (continuation of Deferred@1820092240 after retry RPC@1931766053))) after lookup
16:11:21.923 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] connecting to peer
16:11:21.924 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:11:21.924 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:11:21.924 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:11:22.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:22.617553 18685 raft_consensus.cc:482] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1bce670edc6c40b2bb1584ec3b364e0f)
16:11:22.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:22.617997 18685 raft_consensus.cc:504] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:11:22.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:22.618809 18686 raft_consensus.cc:482] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1bce670edc6c40b2bb1584ec3b364e0f)
16:11:22.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:22.619200 18686 raft_consensus.cc:504] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:11:22.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:22.620553 18685 leader_election.cc:288] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d392ba816c24499a0f7fecd95f88c15 (127.16.234.131:42843), 1bce670edc6c40b2bb1584ec3b364e0f (127.16.234.130:43937)
16:11:22.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:22.621399 18686 leader_election.cc:288] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9315253ea9e464c9416773b52ee00a6 (127.16.234.129:42901), 1bce670edc6c40b2bb1584ec3b364e0f (127.16.234.130:43937)
16:11:22.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:22.630095 18348 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.234.130:43937: connect: Connection refused (error 111)
16:11:22.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:22.632690 17859 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.234.130:43937: connect: Connection refused (error 111)
16:11:22.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:22.636794 18348 leader_election.cc:334] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1bce670edc6c40b2bb1584ec3b364e0f (127.16.234.130:43937): Network error: Client connection negotiation failed: client connection to 127.16.234.130:43937: connect: Connection refused (error 111)
16:11:22.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:22.639236 17859 leader_election.cc:334] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1bce670edc6c40b2bb1584ec3b364e0f (127.16.234.130:43937): Network error: Client connection negotiation failed: client connection to 127.16.234.130:43937: connect: Connection refused (error 111)
16:11:22.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:22.644049 18472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d3c128d101444a59e009013a5c634dc" candidate_uuid: "d9315253ea9e464c9416773b52ee00a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d392ba816c24499a0f7fecd95f88c15" is_pre_election: true
16:11:22.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:22.644687 18472 raft_consensus.cc:2454] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9315253ea9e464c9416773b52ee00a6 in term 1.
16:11:22.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:22.645125 17972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d3c128d101444a59e009013a5c634dc" candidate_uuid: "3d392ba816c24499a0f7fecd95f88c15" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d9315253ea9e464c9416773b52ee00a6" is_pre_election: true
16:11:22.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:22.645680 17972 raft_consensus.cc:2454] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d392ba816c24499a0f7fecd95f88c15 in term 1.
16:11:22.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:22.645795 17857 leader_election.cc:302] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d392ba816c24499a0f7fecd95f88c15, d9315253ea9e464c9416773b52ee00a6; no voters: 1bce670edc6c40b2bb1584ec3b364e0f
16:11:22.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:22.646632 18685 raft_consensus.cc:2781] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 1 FOLLOWER]: Leader pre-election won for term 2
16:11:22.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:22.646732 18347 leader_election.cc:302] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d392ba816c24499a0f7fecd95f88c15, d9315253ea9e464c9416773b52ee00a6; no voters: 1bce670edc6c40b2bb1584ec3b364e0f
16:11:22.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:22.646947 18685 raft_consensus.cc:482] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1bce670edc6c40b2bb1584ec3b364e0f)
16:11:22.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:22.647154 18685 raft_consensus.cc:3037] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 1 FOLLOWER]: Advancing to term 2
16:11:22.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:22.647501 18686 raft_consensus.cc:2781] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 1 FOLLOWER]: Leader pre-election won for term 2
16:11:22.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:22.647799 18686 raft_consensus.cc:482] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1bce670edc6c40b2bb1584ec3b364e0f)
16:11:22.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:22.648113 18686 raft_consensus.cc:3037] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 1 FOLLOWER]: Advancing to term 2
16:11:22.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:22.653330 18685 raft_consensus.cc:504] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:11:22.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:22.654109 18686 raft_consensus.cc:504] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:11:22.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:22.655257 18685 leader_election.cc:288] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [CANDIDATE]: Term 2 election: Requested vote from peers 3d392ba816c24499a0f7fecd95f88c15 (127.16.234.131:42843), 1bce670edc6c40b2bb1584ec3b364e0f (127.16.234.130:43937)
16:11:22.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:22.655877 18686 leader_election.cc:288] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [CANDIDATE]: Term 2 election: Requested vote from peers d9315253ea9e464c9416773b52ee00a6 (127.16.234.129:42901), 1bce670edc6c40b2bb1584ec3b364e0f (127.16.234.130:43937)
16:11:22.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:22.656180 18472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d3c128d101444a59e009013a5c634dc" candidate_uuid: "d9315253ea9e464c9416773b52ee00a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d392ba816c24499a0f7fecd95f88c15"
16:11:22.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:22.656946 18472 raft_consensus.cc:2379] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d9315253ea9e464c9416773b52ee00a6 in current term 2: Already voted for candidate 3d392ba816c24499a0f7fecd95f88c15 in this term.
16:11:22.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:22.657714 17972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d3c128d101444a59e009013a5c634dc" candidate_uuid: "3d392ba816c24499a0f7fecd95f88c15" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d9315253ea9e464c9416773b52ee00a6"
16:11:22.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:22.658365 17972 raft_consensus.cc:2379] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d392ba816c24499a0f7fecd95f88c15 in current term 2: Already voted for candidate d9315253ea9e464c9416773b52ee00a6 in this term.
16:11:22.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:22.659457 17859 leader_election.cc:334] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1bce670edc6c40b2bb1584ec3b364e0f (127.16.234.130:43937): Network error: Client connection negotiation failed: client connection to 127.16.234.130:43937: connect: Connection refused (error 111)
16:11:22.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:22.659811 17859 leader_election.cc:302] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9315253ea9e464c9416773b52ee00a6; no voters: 1bce670edc6c40b2bb1584ec3b364e0f, 3d392ba816c24499a0f7fecd95f88c15
16:11:22.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:22.660399 18685 raft_consensus.cc:2726] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
16:11:22.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:22.660772 18348 leader_election.cc:334] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1bce670edc6c40b2bb1584ec3b364e0f (127.16.234.130:43937): Network error: Client connection negotiation failed: client connection to 127.16.234.130:43937: connect: Connection refused (error 111)
16:11:22.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:22.661090 18348 leader_election.cc:302] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d392ba816c24499a0f7fecd95f88c15; no voters: 1bce670edc6c40b2bb1584ec3b364e0f, d9315253ea9e464c9416773b52ee00a6
16:11:22.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:22.661593 18686 raft_consensus.cc:2726] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
16:11:24.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:24.410744 18701 raft_consensus.cc:482] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
16:11:24.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:24.411113 18701 raft_consensus.cc:504] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:11:24.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:24.412513 18701 leader_election.cc:288] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9315253ea9e464c9416773b52ee00a6 (127.16.234.129:42901), 1bce670edc6c40b2bb1584ec3b364e0f (127.16.234.130:43937)
16:11:24.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:24.413836 17972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d3c128d101444a59e009013a5c634dc" candidate_uuid: "3d392ba816c24499a0f7fecd95f88c15" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d9315253ea9e464c9416773b52ee00a6" is_pre_election: true
16:11:24.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:24.414585 17972 raft_consensus.cc:2454] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d392ba816c24499a0f7fecd95f88c15 in term 2.
16:11:24.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:24.415674 18347 leader_election.cc:302] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d392ba816c24499a0f7fecd95f88c15, d9315253ea9e464c9416773b52ee00a6; no voters: 
16:11:24.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:24.416293 18701 raft_consensus.cc:2781] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 2 FOLLOWER]: Leader pre-election won for term 3
16:11:24.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:24.416535 18701 raft_consensus.cc:482] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
16:11:24.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:24.416836 18701 raft_consensus.cc:3037] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 2 FOLLOWER]: Advancing to term 3
16:11:24.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:24.417678 18348 leader_election.cc:334] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bce670edc6c40b2bb1584ec3b364e0f (127.16.234.130:43937): Network error: Client connection negotiation failed: client connection to 127.16.234.130:43937: connect: Connection refused (error 111)
16:11:24.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:24.421561 18701 raft_consensus.cc:504] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:11:24.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:24.422883 18701 leader_election.cc:288] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [CANDIDATE]: Term 3 election: Requested vote from peers d9315253ea9e464c9416773b52ee00a6 (127.16.234.129:42901), 1bce670edc6c40b2bb1584ec3b364e0f (127.16.234.130:43937)
16:11:24.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:24.423655 17972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d3c128d101444a59e009013a5c634dc" candidate_uuid: "3d392ba816c24499a0f7fecd95f88c15" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d9315253ea9e464c9416773b52ee00a6"
16:11:24.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:24.424099 17972 raft_consensus.cc:3037] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 2 FOLLOWER]: Advancing to term 3
16:11:24.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:24.425731 18348 leader_election.cc:334] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 1bce670edc6c40b2bb1584ec3b364e0f (127.16.234.130:43937): Network error: Client connection negotiation failed: client connection to 127.16.234.130:43937: connect: Connection refused (error 111)
16:11:24.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:24.429064 17972 raft_consensus.cc:2454] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d392ba816c24499a0f7fecd95f88c15 in term 3.
16:11:24.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:24.429854 18347 leader_election.cc:302] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d392ba816c24499a0f7fecd95f88c15, d9315253ea9e464c9416773b52ee00a6; no voters: 1bce670edc6c40b2bb1584ec3b364e0f
16:11:24.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:24.430446 18701 raft_consensus.cc:2781] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 3 FOLLOWER]: Leader election won for term 3
16:11:24.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:24.431239 18701 raft_consensus.cc:686] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 3 LEADER]: Becoming Leader. State: Replica: 3d392ba816c24499a0f7fecd95f88c15, State: Running, Role: LEADER
16:11:24.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:24.432076 18701 consensus_queue.cc:237] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:11:24.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:24.440685 17662 catalog_manager.cc:5495] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 reported cstate change: term changed from 1 to 3, leader changed from 1bce670edc6c40b2bb1584ec3b364e0f (127.16.234.130) to 3d392ba816c24499a0f7fecd95f88c15 (127.16.234.131). New cstate: current_term: 3 leader_uuid: "3d392ba816c24499a0f7fecd95f88c15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } health_report { overall_health: UNKNOWN } } }
16:11:24.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:24.821974 17972 raft_consensus.cc:1261] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 3 FOLLOWER]: Refusing update from remote peer 3d392ba816c24499a0f7fecd95f88c15: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
16:11:24.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:24.823694 18701 consensus_queue.cc:1035] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
16:11:24.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:24.841204 18348 consensus_peers.cc:487] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 -> Peer 1bce670edc6c40b2bb1584ec3b364e0f (127.16.234.130:43937): Couldn't send request to peer 1bce670edc6c40b2bb1584ec3b364e0f. Status: Network error: Client connection negotiation failed: client connection to 127.16.234.130:43937: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
16:11:25.642 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:11:25.642 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937),3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)[L],d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)]}]
16:11:25.643 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=14, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=4369), Traces: [0ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [4ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: connection closed, [4ms] delaying RPC due to: Network error: connection closed, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [28ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [30ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [33ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [34ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [43ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [47ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [49ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [50ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [50ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [63ms] refreshing cache from master, [63ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [68ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [71ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [73ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [74ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [84ms] refreshing cache from master, [84ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [87ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [90ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [92ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [92ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [284ms] refreshing cache from master, [284ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [288ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [291ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [294ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [294ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [643ms] refreshing cache from master, [643ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [647ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [650ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [651ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [651ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [4364ms] refreshing cache from master, [4364ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [4368ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@274916208(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@686091c0 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1775610208 after retry RPC@376111161) -> (continuation of Deferred@1995913747 after retry RPC@332941615) -> (continuation of Deferred@936185223 after retry RPC@1720564576) -> (continuation of Deferred@325465858 after retry RPC@719758382) -> (continuation of Deferred@1820092240 after retry RPC@1931766053) -> (continuation of Deferred@619282930 after retry RPC@1805130431), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@31e9c516 -> wakeup thread Time-limited test -> (continuation of Deferred@1775610208 after retry RPC@376111161) -> (continuation of Deferred@1995913747 after retry RPC@332941615) -> (continuation of Deferred@936185223 after retry RPC@1720564576) -> (continuation of Deferred@325465858 after retry RPC@719758382) -> (continuation of Deferred@1820092240 after retry RPC@1931766053) -> (continuation of Deferred@619282930 after retry RPC@1805130431))) after lookup
16:11:25.644 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)] connecting to peer
16:11:25.646 [DEBUG - kudu-nio-0] (Connection.java:757) [peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)] Successfully connected to peer
16:11:25.667 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0x1a12cad8, L:/127.0.0.1:51428 - R:/127.16.234.131:42843] using TOKEN/null
16:11:25.712 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.16.234.130:43937
16:11:25.713 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

16:11:25.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:25.715029 17322 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu
16:11:25.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu
16:11:25.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/wal
16:11:25.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/data
16:11:25.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
16:11:25.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
16:11:25.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
16:11:25.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:11:25.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:11:25.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:11:25.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/logs
16:11:25.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/data/info.pb
16:11:25.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:11:25.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
16:11:25.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
16:11:25.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
16:11:25.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
16:11:25.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:11:25.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:11:25.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:11:25.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
16:11:25.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
16:11:25.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.234.130:43937
16:11:25.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.16.234.130
16:11:25.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.16.234.190:33901,127.16.234.189:32987,127.16.234.188:38065
16:11:25.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=37863
16:11:25.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.234.130
16:11:25.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.234.148:43519
16:11:25.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:11:25.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:11:25.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
16:11:26.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:26.037211 18753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
16:11:26.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:26.037703 18753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
16:11:26.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:26.038173 18753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
16:11:26.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:26.069221 18753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
16:11:26.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:26.070039 18753 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.234.130
16:11:26.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:26.104967 18753 tablet_server_runner.cc:78] Tablet server non-default flags:
16:11:26.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:11:26.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.234.148:43519
16:11:26.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:11:26.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
16:11:26.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/data
16:11:26.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/wal
16:11:26.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:11:26.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:11:26.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.234.130:43937
16:11:26.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
16:11:26.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:11:26.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:11:26.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/data/info.pb
16:11:26.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.234.130
16:11:26.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=37863
16:11:26.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.16.234.190:33901,127.16.234.189:32987,127.16.234.188:38065
16:11:26.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
16:11:26.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:11:26.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
16:11:26.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
16:11:26.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:11:26.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.16.234.130
16:11:26.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/logs
16:11:26.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:11:26.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
16:11:26.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
16:11:26.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
16:11:26.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
16:11:26.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
16:11:26.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
16:11:26.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 15:12:52 UTC on ccycloud.quasar-tzrfpd.root.comops.site
16:11:26.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4580143
16:11:26.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
16:11:26.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:26.106279 18753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
16:11:26.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:26.107896 18753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
16:11:26.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:26.126957 18769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:11:27.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:26.127628 18765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:11:27.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:26.127911 18773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:11:27.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:27.232944 18771 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
16:11:27.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:27.233095 18753 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
16:11:27.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:27.236780 18753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
16:11:27.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:27.239388 18753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
16:11:27.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:27.240797 18753 hybrid_clock.cc:648] HybridClock initialized: now 1786464687240764 us; error 52 us; skew 500 ppm
16:11:27.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:27.241665 18753 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
16:11:27.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:27.247767 18753 webserver.cc:460] Webserver started at http://127.16.234.130:37863/ using document root <none> and password file <none>
16:11:27.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:27.248643 18753 fs_manager.cc:359] Metadata directory not provided
16:11:27.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:27.248839 18753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
16:11:27.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:27.256523 18753 fs_manager.cc:709] Time spent opening directory manager: real 0.005s	user 0.005s	sys 0.000s
16:11:27.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:27.262357 18797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
16:11:27.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:27.263419 18753 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
16:11:27.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:27.263720 18753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/data,/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/wal
16:11:27.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "1bce670edc6c40b2bb1584ec3b364e0f"
16:11:27.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 16:11:16 on dist-test-slave-zj8q"
16:11:27.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:27.266077 18753 fs_report.cc:383] FS layout report
16:11:27.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
16:11:27.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/wal
16:11:27.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/wal
16:11:27.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/data/data
16:11:27.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
16:11:27.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
16:11:27.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
16:11:27.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
16:11:27.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
16:11:27.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
16:11:27.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
16:11:27.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
16:11:27.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
16:11:27.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
16:11:27.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
16:11:27.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:27.339808 18753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
16:11:27.341 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:27.341324 18753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
16:11:27.341 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:27.341805 18753 kserver.cc:163] Server-wide thread pool size limit: 3276
16:11:27.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:27.344691 18753 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
16:11:27.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:27.350653 18811 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
16:11:27.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:27.358500 18753 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
16:11:27.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:27.358729 18753 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s	user 0.000s	sys 0.002s
16:11:27.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:27.359045 18753 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
16:11:27.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:27.364180 18753 ts_tablet_manager.cc:610] Registered 1 tablets
16:11:27.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:27.364360 18753 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s	user 0.002s	sys 0.001s
16:11:27.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:27.364674 18811 tablet_bootstrap.cc:492] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f: Bootstrap starting.
16:11:27.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:27.416693 18811 log.cc:824] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f: Log is configured to *not* fsync() on all Append() calls
16:11:27.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:27.444993 18811 tablet_bootstrap.cc:492] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
16:11:27.446 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:27.445909 18811 tablet_bootstrap.cc:492] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f: Bootstrap complete.
16:11:27.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:27.447218 18811 ts_tablet_manager.cc:1397] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f: Time spent bootstrapping tablet: real 0.083s	user 0.056s	sys 0.024s
16:11:27.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:27.468487 18811 raft_consensus.cc:348] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:11:27.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:27.471361 18811 raft_consensus.cc:729] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bce670edc6c40b2bb1584ec3b364e0f, State: Initialized, Role: FOLLOWER
16:11:27.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:27.472393 18811 consensus_queue.cc:260] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:11:27.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:27.475497 18811 ts_tablet_manager.cc:1428] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f: Time spent starting tablet: real 0.028s	user 0.020s	sys 0.008s
16:11:27.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:27.533283 18753 rpc_server.cc:307] RPC server started. Bound to: 127.16.234.130:43937
16:11:27.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:27.533514 18980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.234.130:43937 every 8 connection(s)
16:11:27.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:27.535921 18753 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/data/info.pb
16:11:27.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:27.542762 17322 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu as pid 18753
16:11:27.543 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
16:11:27.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:27.579528 18981 heartbeater.cc:347] Connected to a master server at 127.16.234.188:38065
16:11:27.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:27.580024 18981 heartbeater.cc:464] Registering TS with master...
16:11:27.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:27.581411 18981 heartbeater.cc:511] Master 127.16.234.188:38065 requested a full tablet report, sending...
16:11:27.587 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 6 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304, tablet=7d3c128d101444a59e009013a5c634dc, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
16:11:27.589 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.16.234.131:42843
16:11:27.589 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

16:11:27.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:27.586486 17662 ts_manager.cc:194] Re-registered known tserver with Master: 1bce670edc6c40b2bb1584ec3b364e0f (127.16.234.130:43937)
16:11:27.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:27.589766 17322 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu with pid 18318
16:11:27.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:27.589946 17662 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.234.130:33545
16:11:27.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:27.600425 18988 heartbeater.cc:347] Connected to a master server at 127.16.234.190:33901
16:11:27.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:27.600854 18988 heartbeater.cc:464] Registering TS with master...
16:11:27.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:27.601730 18988 heartbeater.cc:511] Master 127.16.234.190:33901 requested a full tablet report, sending...
16:11:27.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:27.605849 17402 ts_manager.cc:194] Re-registered known tserver with Master: 1bce670edc6c40b2bb1584ec3b364e0f (127.16.234.130:43937)
16:11:27.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:27.606823 18984 heartbeater.cc:347] Connected to a master server at 127.16.234.189:32987
16:11:27.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:27.607118 18984 heartbeater.cc:464] Registering TS with master...
16:11:27.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:27.607695 18984 heartbeater.cc:511] Master 127.16.234.189:32987 requested a full tablet report, sending...
16:11:27.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:27.610428 17537 ts_manager.cc:194] Re-registered known tserver with Master: 1bce670edc6c40b2bb1584ec3b364e0f (127.16.234.130:43937)
16:11:27.629 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
16:11:27.629 [DEBUG - kudu-nio-0] (Connection.java:239) [peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)] handling channelInactive
16:11:27.629 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)] cleaning up while in state READY due to: connection closed
16:11:27.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:27.630484 19004 negotiation.cc:337] Failed RPC negotiation. Trace:
16:11:27.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:11:27.576424 (+     0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.234.131:37139 (local address 127.16.234.130:43937)
16:11:27.630 [DEBUG - Time-limited test] (Connection.java:748) [peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)] connecting to peer
16:11:27.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:11:27.597703 (+ 21279us) server_negotiation.cc:206] Beginning negotiation
16:11:27.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:11:27.597734 (+    31us) server_negotiation.cc:399] Waiting for connection header
16:11:27.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:11:27.597790 (+    56us) server_negotiation.cc:407] Connection header received
16:11:27.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:11:27.597991 (+   201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
16:11:27.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:11:27.598025 (+    34us) server_negotiation.cc:461] Received NEGOTIATE request from client
16:11:27.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:11:27.599885 (+  1860us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
16:11:27.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:11:27.600051 (+   166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
16:11:27.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:11:27.628552 (+ 28501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.234.131:37139: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
16:11:27.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":20757,"thread_start_us":20369,"threads_started":1}
16:11:27.632 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843
16:11:27.632 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843
16:11:27.632 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843
16:11:27.641 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:11:27.641 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937),3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)[L],d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)]}]
16:11:27.642 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=2, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=12), Traces: [0ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [2ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [2ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [10ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@478312488(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@5b5c7fd2 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3fed6a3 -> wakeup thread Time-limited test)) after lookup
16:11:27.643 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)] connecting to peer
16:11:27.644 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843
16:11:27.644 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843
16:11:27.645 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843
16:11:27.661 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:11:27.661 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937),3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)[L],d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)]}]
16:11:27.662 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=4, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=32), Traces: [0ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [2ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [2ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [10ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [12ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [14ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [15ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [30ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@478312488(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@5b5c7fd2 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1924304368 after retry RPC@118232779), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3fed6a3 -> wakeup thread Time-limited test -> (continuation of Deferred@1924304368 after retry RPC@118232779))) after lookup
16:11:27.663 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)] connecting to peer
16:11:27.664 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843
16:11:27.664 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843
16:11:27.664 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843
16:11:27.702 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:11:27.702 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937),3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)[L],d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)]}]
16:11:27.703 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=6, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=73), Traces: [0ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [2ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [2ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [10ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [12ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [14ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [15ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [30ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [33ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [34ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [35ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [67ms] refreshing cache from master, [67ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [71ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@478312488(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@5b5c7fd2 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1924304368 after retry RPC@118232779) -> (continuation of Deferred@1089271174 after retry RPC@637477709), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3fed6a3 -> wakeup thread Time-limited test -> (continuation of Deferred@1924304368 after retry RPC@118232779) -> (continuation of Deferred@1089271174 after retry RPC@637477709))) after lookup
16:11:27.704 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)] connecting to peer
16:11:27.706 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843
16:11:27.706 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843
16:11:27.706 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843
16:11:27.822 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:11:27.822 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937),3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)[L],d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)]}]
16:11:27.823 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=8, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=193), Traces: [0ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [2ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [2ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [10ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [12ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [14ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [15ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [30ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [33ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [34ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [35ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [67ms] refreshing cache from master, [67ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [71ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [74ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [76ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [76ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [187ms] refreshing cache from master, [187ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [191ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@478312488(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@5b5c7fd2 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1924304368 after retry RPC@118232779) -> (continuation of Deferred@1089271174 after retry RPC@637477709) -> (continuation of Deferred@174360944 after retry RPC@1457333818), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3fed6a3 -> wakeup thread Time-limited test -> (continuation of Deferred@1924304368 after retry RPC@118232779) -> (continuation of Deferred@1089271174 after retry RPC@637477709) -> (continuation of Deferred@174360944 after retry RPC@1457333818))) after lookup
16:11:27.824 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)] connecting to peer
16:11:27.825 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843
16:11:27.826 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843
16:11:27.826 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843
16:11:28.121 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:11:28.121 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937),3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)[L],d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)]}]
16:11:28.122 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=10, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=492), Traces: [0ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [2ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [2ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [10ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [12ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [14ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [15ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [30ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [33ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [34ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [35ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [67ms] refreshing cache from master, [67ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [71ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [74ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [76ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [76ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [187ms] refreshing cache from master, [187ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [191ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [193ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [196ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [196ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [486ms] refreshing cache from master, [486ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [490ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@478312488(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@5b5c7fd2 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1924304368 after retry RPC@118232779) -> (continuation of Deferred@1089271174 after retry RPC@637477709) -> (continuation of Deferred@174360944 after retry RPC@1457333818) -> (continuation of Deferred@1947498888 after retry RPC@938836649), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3fed6a3 -> wakeup thread Time-limited test -> (continuation of Deferred@1924304368 after retry RPC@118232779) -> (continuation of Deferred@1089271174 after retry RPC@637477709) -> (continuation of Deferred@174360944 after retry RPC@1457333818) -> (continuation of Deferred@1947498888 after retry RPC@938836649))) after lookup
16:11:28.123 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)] connecting to peer
16:11:28.126 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843
16:11:28.126 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843
16:11:28.126 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843
16:11:28.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:28.600026 18981 heartbeater.cc:503] Master 127.16.234.188:38065 was elected leader, sending a full tablet report...
16:11:28.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:28.644383 19008 raft_consensus.cc:482] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3d392ba816c24499a0f7fecd95f88c15)
16:11:28.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:28.644785 19008 raft_consensus.cc:504] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:11:28.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:28.646474 19008 leader_election.cc:288] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d392ba816c24499a0f7fecd95f88c15 (127.16.234.131:42843), 1bce670edc6c40b2bb1584ec3b364e0f (127.16.234.130:43937)
16:11:28.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:28.648950 17857 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.234.131:42843: connect: Connection refused (error 111) [suppressed 1 similar messages]
16:11:28.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:28.653100 17857 leader_election.cc:334] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 3d392ba816c24499a0f7fecd95f88c15 (127.16.234.131:42843): Network error: Client connection negotiation failed: client connection to 127.16.234.131:42843: connect: Connection refused (error 111)
16:11:28.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:28.660554 18908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d3c128d101444a59e009013a5c634dc" candidate_uuid: "d9315253ea9e464c9416773b52ee00a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: false dest_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" is_pre_election: true
16:11:28.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:28.661303 18908 raft_consensus.cc:2454] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9315253ea9e464c9416773b52ee00a6 in term 1.
16:11:28.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:28.662551 17859 leader_election.cc:302] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1bce670edc6c40b2bb1584ec3b364e0f, d9315253ea9e464c9416773b52ee00a6; no voters: 3d392ba816c24499a0f7fecd95f88c15
16:11:28.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:28.663148 19008 raft_consensus.cc:2781] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 3 FOLLOWER]: Leader pre-election won for term 4
16:11:28.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:28.663404 19008 raft_consensus.cc:482] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3d392ba816c24499a0f7fecd95f88c15)
16:11:28.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:28.663686 19008 raft_consensus.cc:3037] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 3 FOLLOWER]: Advancing to term 4
16:11:28.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:28.667718 19008 raft_consensus.cc:504] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:11:28.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:28.669104 19008 leader_election.cc:288] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [CANDIDATE]: Term 4 election: Requested vote from peers 3d392ba816c24499a0f7fecd95f88c15 (127.16.234.131:42843), 1bce670edc6c40b2bb1584ec3b364e0f (127.16.234.130:43937)
16:11:28.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:28.670105 18908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d3c128d101444a59e009013a5c634dc" candidate_uuid: "d9315253ea9e464c9416773b52ee00a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: false dest_uuid: "1bce670edc6c40b2bb1584ec3b364e0f"
16:11:28.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:28.670666 18908 raft_consensus.cc:3037] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [term 1 FOLLOWER]: Advancing to term 4
16:11:28.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:28.672030 17857 leader_election.cc:334] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 3d392ba816c24499a0f7fecd95f88c15 (127.16.234.131:42843): Network error: Client connection negotiation failed: client connection to 127.16.234.131:42843: connect: Connection refused (error 111)
16:11:28.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:28.677383 18908 raft_consensus.cc:2454] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9315253ea9e464c9416773b52ee00a6 in term 4.
16:11:28.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:28.678243 17859 leader_election.cc:302] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1bce670edc6c40b2bb1584ec3b364e0f, d9315253ea9e464c9416773b52ee00a6; no voters: 3d392ba816c24499a0f7fecd95f88c15
16:11:28.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:28.678841 19008 raft_consensus.cc:2781] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 4 FOLLOWER]: Leader election won for term 4
16:11:28.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:28.679903 19008 raft_consensus.cc:686] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 4 LEADER]: Becoming Leader. State: Replica: d9315253ea9e464c9416773b52ee00a6, State: Running, Role: LEADER
16:11:28.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:28.681219 19008 consensus_queue.cc:237] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 3.8, Last appended by leader: 8, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:11:28.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:28.690229 17662 catalog_manager.cc:5495] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 reported cstate change: term changed from 3 to 4, leader changed from 3d392ba816c24499a0f7fecd95f88c15 (127.16.234.131) to d9315253ea9e464c9416773b52ee00a6 (127.16.234.129). New cstate: current_term: 4 leader_uuid: "d9315253ea9e464c9416773b52ee00a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } health_report { overall_health: UNKNOWN } } }
16:11:29.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:29.097975 17857 consensus_peers.cc:487] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 -> Peer 3d392ba816c24499a0f7fecd95f88c15 (127.16.234.131:42843): Couldn't send request to peer 3d392ba816c24499a0f7fecd95f88c15. Status: Network error: Client connection negotiation failed: client connection to 127.16.234.131:42843: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
16:11:29.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:29.248301 18908 raft_consensus.cc:1261] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [term 4 FOLLOWER]: Refusing update from remote peer d9315253ea9e464c9416773b52ee00a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 9. (index mismatch)
16:11:29.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:29.249769 19008 consensus_queue.cc:1035] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 4, Time since last communication: 0.000s
16:11:29.401 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:11:29.402 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937),3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843),d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)[L]]}]
16:11:29.402 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=12, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=1772), Traces: [0ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [2ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [2ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [10ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [12ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [14ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [15ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [30ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [33ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [34ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [35ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [67ms] refreshing cache from master, [67ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [71ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [74ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [76ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [76ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [187ms] refreshing cache from master, [187ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [191ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [193ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [196ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [196ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [486ms] refreshing cache from master, [486ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [490ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [493ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [496ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [497ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [1766ms] refreshing cache from master, [1766ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [1770ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@478312488(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@5b5c7fd2 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1924304368 after retry RPC@118232779) -> (continuation of Deferred@1089271174 after retry RPC@637477709) -> (continuation of Deferred@174360944 after retry RPC@1457333818) -> (continuation of Deferred@1947498888 after retry RPC@938836649) -> (continuation of Deferred@649255878 after retry RPC@1028257139), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3fed6a3 -> wakeup thread Time-limited test -> (continuation of Deferred@1924304368 after retry RPC@118232779) -> (continuation of Deferred@1089271174 after retry RPC@637477709) -> (continuation of Deferred@174360944 after retry RPC@1457333818) -> (continuation of Deferred@1947498888 after retry RPC@938836649) -> (continuation of Deferred@649255878 after retry RPC@1028257139))) after lookup
16:11:29.404 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)] connecting to peer
16:11:29.405 [DEBUG - kudu-nio-1] (Connection.java:757) [peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)] Successfully connected to peer
16:11:29.431 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0xe83b66af, L:/127.0.0.1:39384 - R:/127.16.234.129:42901] using TOKEN/null
16:11:29.479 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.16.234.131:42843
16:11:29.480 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

16:11:29.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:29.481997 17322 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu
16:11:29.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu
16:11:29.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-2/wal
16:11:29.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-2/data
16:11:29.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
16:11:29.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
16:11:29.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
16:11:29.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:11:29.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:11:29.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:11:29.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-2/logs
16:11:29.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-2/data/info.pb
16:11:29.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:11:29.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
16:11:29.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
16:11:29.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
16:11:29.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
16:11:29.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:11:29.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:11:29.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:11:29.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
16:11:29.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
16:11:29.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.234.131:42843
16:11:29.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.16.234.131
16:11:29.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.16.234.190:33901,127.16.234.189:32987,127.16.234.188:38065
16:11:29.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=35277
16:11:29.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.234.131
16:11:29.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.234.148:43519
16:11:29.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:11:29.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:11:29.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
16:11:29.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:29.805779 19055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
16:11:29.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:29.806303 19055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
16:11:29.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:29.806849 19055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
16:11:29.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:29.837515 19055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
16:11:29.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:29.838362 19055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.234.131
16:11:29.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:29.873565 19055 tablet_server_runner.cc:78] Tablet server non-default flags:
16:11:29.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:11:29.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.234.148:43519
16:11:29.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:11:29.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
16:11:29.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-2/data
16:11:29.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-2/wal
16:11:29.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:11:29.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:11:29.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.234.131:42843
16:11:29.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
16:11:29.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:11:29.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:11:29.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-2/data/info.pb
16:11:29.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.234.131
16:11:29.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=35277
16:11:29.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.16.234.190:33901,127.16.234.189:32987,127.16.234.188:38065
16:11:29.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
16:11:29.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:11:29.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
16:11:29.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
16:11:29.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:11:29.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.16.234.131
16:11:29.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-2/logs
16:11:29.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:11:29.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
16:11:29.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
16:11:29.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
16:11:29.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
16:11:29.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
16:11:29.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
16:11:29.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 15:12:52 UTC on ccycloud.quasar-tzrfpd.root.comops.site
16:11:29.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4580143
16:11:29.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
16:11:29.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:29.874756 19055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
16:11:29.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:29.876284 19055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
16:11:29.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:29.894879 19068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:11:30.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:29.894881 19069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:11:30.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:29.896497 19055 server_base.cc:1034] running on GCE node
16:11:30.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:29.895030 19074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:11:30.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:30.997172 19055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
16:11:30.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:30.999508 19055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
16:11:31.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:31.000900 19055 hybrid_clock.cc:648] HybridClock initialized: now 1786464691000863 us; error 38 us; skew 500 ppm
16:11:31.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:31.001739 19055 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
16:11:31.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:31.012847 19055 webserver.cc:460] Webserver started at http://127.16.234.131:35277/ using document root <none> and password file <none>
16:11:31.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:31.013852 19055 fs_manager.cc:359] Metadata directory not provided
16:11:31.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:31.014101 19055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
16:11:31.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:31.022140 19055 fs_manager.cc:709] Time spent opening directory manager: real 0.005s	user 0.006s	sys 0.001s
16:11:31.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:31.026870 19100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
16:11:31.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:31.027900 19055 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.002s	sys 0.002s
16:11:31.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:31.028251 19055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-2/data,/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-2/wal
16:11:31.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "3d392ba816c24499a0f7fecd95f88c15"
16:11:31.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 16:11:18 on dist-test-slave-zj8q"
16:11:31.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:31.031113 19055 fs_report.cc:383] FS layout report
16:11:31.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
16:11:31.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-2/wal
16:11:31.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-2/wal
16:11:31.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-2/data/data
16:11:31.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
16:11:31.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
16:11:31.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
16:11:31.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
16:11:31.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
16:11:31.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
16:11:31.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
16:11:31.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
16:11:31.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
16:11:31.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
16:11:31.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
16:11:31.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:31.078940 19055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
16:11:31.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:31.080732 19055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
16:11:31.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:31.081358 19055 kserver.cc:163] Server-wide thread pool size limit: 3276
16:11:31.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:31.084043 19055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
16:11:31.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:31.090793 19111 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
16:11:31.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:31.098909 19055 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
16:11:31.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:31.099159 19055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s	user 0.000s	sys 0.002s
16:11:31.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:31.099499 19055 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
16:11:31.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:31.104323 19055 ts_tablet_manager.cc:610] Registered 1 tablets
16:11:31.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:31.104558 19055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s	user 0.000s	sys 0.003s
16:11:31.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:31.104923 19111 tablet_bootstrap.cc:492] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15: Bootstrap starting.
16:11:31.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:31.171267 19111 log.cc:824] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15: Log is configured to *not* fsync() on all Append() calls
16:11:31.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:31.220216 19111 tablet_bootstrap.cc:492] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=6 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
16:11:31.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:31.221344 19111 tablet_bootstrap.cc:492] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15: Bootstrap complete.
16:11:31.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:31.223218 19111 ts_tablet_manager.cc:1397] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15: Time spent bootstrapping tablet: real 0.119s	user 0.071s	sys 0.038s
16:11:31.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:31.246186 19111 raft_consensus.cc:348] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:11:31.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:31.248778 19111 raft_consensus.cc:729] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d392ba816c24499a0f7fecd95f88c15, State: Initialized, Role: FOLLOWER
16:11:31.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:31.249835 19111 consensus_queue.cc:260] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 3.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:11:31.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:31.254452 19111 ts_tablet_manager.cc:1428] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15: Time spent starting tablet: real 0.031s	user 0.022s	sys 0.008s
16:11:31.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:31.276212 19055 rpc_server.cc:307] RPC server started. Bound to: 127.16.234.131:42843
16:11:31.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:31.276358 19276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.234.131:42843 every 8 connection(s)
16:11:31.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:31.279044 19055 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-2/data/info.pb
16:11:31.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:31.289630 17322 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu as pid 19055
16:11:31.291 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
16:11:31.326 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 9 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304, tablet=7d3c128d101444a59e009013a5c634dc, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
16:11:31.327 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.16.234.129:42901
16:11:31.328 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

16:11:31.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:31.328572 17322 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu with pid 17790
16:11:31.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:31.355159 19284 heartbeater.cc:347] Connected to a master server at 127.16.234.189:32987
16:11:31.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:31.355681 19284 heartbeater.cc:464] Registering TS with master...
16:11:31.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:31.356897 19284 heartbeater.cc:511] Master 127.16.234.189:32987 requested a full tablet report, sending...
16:11:31.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:31.361680 17537 ts_manager.cc:194] Re-registered known tserver with Master: 3d392ba816c24499a0f7fecd95f88c15 (127.16.234.131:42843)
16:11:31.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:31.363042 19286 heartbeater.cc:347] Connected to a master server at 127.16.234.190:33901
16:11:31.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:31.363401 19286 heartbeater.cc:464] Registering TS with master...
16:11:31.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:31.364207 19286 heartbeater.cc:511] Master 127.16.234.190:33901 requested a full tablet report, sending...
16:11:31.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:31.364778 19283 heartbeater.cc:347] Connected to a master server at 127.16.234.188:38065
16:11:31.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:31.365123 19283 heartbeater.cc:464] Registering TS with master...
16:11:31.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:31.365944 19283 heartbeater.cc:511] Master 127.16.234.188:38065 requested a full tablet report, sending...
16:11:31.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:31.367120 17402 ts_manager.cc:194] Re-registered known tserver with Master: 3d392ba816c24499a0f7fecd95f88c15 (127.16.234.131:42843)
16:11:31.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:31.368903 17662 ts_manager.cc:194] Re-registered known tserver with Master: 3d392ba816c24499a0f7fecd95f88c15 (127.16.234.131:42843)
16:11:31.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:31.372004 17662 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.234.131:47367
16:11:31.382 [DEBUG - kudu-nio-1] (Connection.java:239) [peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)] handling channelInactive
16:11:31.382 [DEBUG - kudu-nio-1] (Connection.java:709) [peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)] cleaning up while in state READY due to: connection closed
16:11:31.383 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
16:11:31.384 [DEBUG - Time-limited test] (Connection.java:748) [peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)] connecting to peer
16:11:31.385 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901
16:11:31.386 [DEBUG - kudu-nio-0] (Connection.java:709) [peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)] cleaning up while in state CONNECTING due to: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901
16:11:31.386 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901
16:11:31.401 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:11:31.401 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937),3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843),d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)[L]]}]
16:11:31.402 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=2, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=18), Traces: [0ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [3ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [3ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [14ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [18ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@1267402690(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@2443ab6 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4ba6c1b8 -> wakeup thread Time-limited test)) after lookup
16:11:31.402 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)] connecting to peer
16:11:31.403 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901
16:11:31.403 [DEBUG - kudu-nio-1] (Connection.java:709) [peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)] cleaning up while in state CONNECTING due to: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901
16:11:31.404 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901
16:11:31.420 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:11:31.421 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937),3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843),d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)[L]]}]
16:11:31.421 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=4, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=38), Traces: [0ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [3ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [3ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [14ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [18ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [19ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [21ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [21ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [34ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [37ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@1267402690(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@2443ab6 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1805664114 after retry RPC@1250009179), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4ba6c1b8 -> wakeup thread Time-limited test -> (continuation of Deferred@1805664114 after retry RPC@1250009179))) after lookup
16:11:31.422 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)] connecting to peer
16:11:31.423 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901
16:11:31.423 [DEBUG - kudu-nio-0] (Connection.java:709) [peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)] cleaning up while in state CONNECTING due to: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901
16:11:31.424 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901
16:11:31.440 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:11:31.440 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937),3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843),d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)[L]]}]
16:11:31.441 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=6, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=57), Traces: [0ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [3ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [3ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [14ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [18ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [19ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [21ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [21ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [34ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [37ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [39ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [41ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [41ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [53ms] refreshing cache from master, [53ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [57ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@1267402690(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@2443ab6 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1805664114 after retry RPC@1250009179) -> (continuation of Deferred@1760216081 after retry RPC@799945263), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4ba6c1b8 -> wakeup thread Time-limited test -> (continuation of Deferred@1805664114 after retry RPC@1250009179) -> (continuation of Deferred@1760216081 after retry RPC@799945263))) after lookup
16:11:31.441 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)] connecting to peer
16:11:31.442 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901
16:11:31.442 [DEBUG - kudu-nio-1] (Connection.java:709) [peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)] cleaning up while in state CONNECTING due to: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901
16:11:31.443 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901
16:11:31.461 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:11:31.461 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937),3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843),d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)[L]]}]
16:11:31.461 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=8, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=78), Traces: [0ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [3ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [3ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [14ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [18ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [19ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [21ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [21ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [34ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [37ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [39ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [41ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [41ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [53ms] refreshing cache from master, [53ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [57ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [58ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [59ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [60ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [74ms] refreshing cache from master, [74ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [77ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@1267402690(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@2443ab6 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1805664114 after retry RPC@1250009179) -> (continuation of Deferred@1760216081 after retry RPC@799945263) -> (continuation of Deferred@169222441 after retry RPC@1675867567), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4ba6c1b8 -> wakeup thread Time-limited test -> (continuation of Deferred@1805664114 after retry RPC@1250009179) -> (continuation of Deferred@1760216081 after retry RPC@799945263) -> (continuation of Deferred@169222441 after retry RPC@1675867567))) after lookup
16:11:31.463 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)] connecting to peer
16:11:31.464 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901
16:11:31.465 [DEBUG - kudu-nio-0] (Connection.java:709) [peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)] cleaning up while in state CONNECTING due to: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901
16:11:31.465 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901
16:11:31.980 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:11:31.981 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937),3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843),d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)[L]]}]
16:11:31.981 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=10, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=597), Traces: [0ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [3ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [3ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [14ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [18ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [19ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [21ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [21ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [34ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [37ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [39ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [41ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [41ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [53ms] refreshing cache from master, [53ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [57ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [58ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [59ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [60ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [74ms] refreshing cache from master, [74ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [77ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [79ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [82ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [82ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [593ms] refreshing cache from master, [593ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [597ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@1267402690(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@2443ab6 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1805664114 after retry RPC@1250009179) -> (continuation of Deferred@1760216081 after retry RPC@799945263) -> (continuation of Deferred@169222441 after retry RPC@1675867567) -> (continuation of Deferred@228127886 after retry RPC@1932031583), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4ba6c1b8 -> wakeup thread Time-limited test -> (continuation of Deferred@1805664114 after retry RPC@1250009179) -> (continuation of Deferred@1760216081 after retry RPC@799945263) -> (continuation of Deferred@169222441 after retry RPC@1675867567) -> (continuation of Deferred@228127886 after retry RPC@1932031583))) after lookup
16:11:31.982 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)] connecting to peer
16:11:31.983 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901
16:11:31.983 [DEBUG - kudu-nio-1] (Connection.java:709) [peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)] cleaning up while in state CONNECTING due to: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901
16:11:31.983 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901
16:11:32.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:32.375744 19283 heartbeater.cc:503] Master 127.16.234.188:38065 was elected leader, sending a full tablet report...
16:11:32.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:32.588594 19303 raft_consensus.cc:482] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d9315253ea9e464c9416773b52ee00a6)
16:11:32.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:32.588985 19303 raft_consensus.cc:504] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:11:32.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:32.591215 19303 leader_election.cc:288] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d392ba816c24499a0f7fecd95f88c15 (127.16.234.131:42843), d9315253ea9e464c9416773b52ee00a6 (127.16.234.129:42901)
16:11:32.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:32.593545 18804 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.234.129:42901: connect: Connection refused (error 111)
16:11:32.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:32.596838 18804 leader_election.cc:334] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer d9315253ea9e464c9416773b52ee00a6 (127.16.234.129:42901): Network error: Client connection negotiation failed: client connection to 127.16.234.129:42901: connect: Connection refused (error 111)
16:11:32.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:32.604048 19202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d3c128d101444a59e009013a5c634dc" candidate_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" candidate_term: 5 candidate_status { last_received { term: 4 index: 12 } } ignore_live_leader: false dest_uuid: "3d392ba816c24499a0f7fecd95f88c15" is_pre_election: true
16:11:32.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:32.604780 19202 raft_consensus.cc:2454] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bce670edc6c40b2bb1584ec3b364e0f in term 3.
16:11:32.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:32.605993 18801 leader_election.cc:302] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1bce670edc6c40b2bb1584ec3b364e0f, 3d392ba816c24499a0f7fecd95f88c15; no voters: d9315253ea9e464c9416773b52ee00a6
16:11:32.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:32.606675 19303 raft_consensus.cc:2781] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [term 4 FOLLOWER]: Leader pre-election won for term 5
16:11:32.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:32.606964 19303 raft_consensus.cc:482] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [term 4 FOLLOWER]: Starting leader election (detected failure of leader d9315253ea9e464c9416773b52ee00a6)
16:11:32.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:32.607261 19303 raft_consensus.cc:3037] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [term 4 FOLLOWER]: Advancing to term 5
16:11:32.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:32.612288 19303 raft_consensus.cc:504] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:11:32.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:32.613654 19303 leader_election.cc:288] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [CANDIDATE]: Term 5 election: Requested vote from peers 3d392ba816c24499a0f7fecd95f88c15 (127.16.234.131:42843), d9315253ea9e464c9416773b52ee00a6 (127.16.234.129:42901)
16:11:32.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:32.614393 19202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d3c128d101444a59e009013a5c634dc" candidate_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" candidate_term: 5 candidate_status { last_received { term: 4 index: 12 } } ignore_live_leader: false dest_uuid: "3d392ba816c24499a0f7fecd95f88c15"
16:11:32.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:32.614878 19202 raft_consensus.cc:3037] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 3 FOLLOWER]: Advancing to term 5
16:11:32.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:32.616767 18804 leader_election.cc:334] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer d9315253ea9e464c9416773b52ee00a6 (127.16.234.129:42901): Network error: Client connection negotiation failed: client connection to 127.16.234.129:42901: connect: Connection refused (error 111)
16:11:32.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:32.621600 19202 raft_consensus.cc:2454] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bce670edc6c40b2bb1584ec3b364e0f in term 5.
16:11:32.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:32.622444 18801 leader_election.cc:302] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1bce670edc6c40b2bb1584ec3b364e0f, 3d392ba816c24499a0f7fecd95f88c15; no voters: d9315253ea9e464c9416773b52ee00a6
16:11:32.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:32.623093 19303 raft_consensus.cc:2781] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [term 5 FOLLOWER]: Leader election won for term 5
16:11:32.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:32.624054 19303 raft_consensus.cc:686] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [term 5 LEADER]: Becoming Leader. State: Replica: 1bce670edc6c40b2bb1584ec3b364e0f, State: Running, Role: LEADER
16:11:32.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:32.624907 19303 consensus_queue.cc:237] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 4.12, Last appended by leader: 12, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:11:32.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:32.634327 17662 catalog_manager.cc:5495] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f reported cstate change: term changed from 4 to 5, leader changed from d9315253ea9e464c9416773b52ee00a6 (127.16.234.129) to 1bce670edc6c40b2bb1584ec3b364e0f (127.16.234.130). New cstate: current_term: 5 leader_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } health_report { overall_health: HEALTHY } } }
16:11:33.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:33.102000 18804 consensus_peers.cc:487] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f -> Peer d9315253ea9e464c9416773b52ee00a6 (127.16.234.129:42901): Couldn't send request to peer d9315253ea9e464c9416773b52ee00a6. Status: Network error: Client connection negotiation failed: client connection to 127.16.234.129:42901: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
16:11:33.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:33.102703 19202 raft_consensus.cc:1261] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 5 FOLLOWER]: Refusing update from remote peer 1bce670edc6c40b2bb1584ec3b364e0f: Log matching property violated. Preceding OpId in replica: term: 3 index: 8. Preceding OpId from leader: term: 5 index: 13. (index mismatch)
16:11:33.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:33.104120 19303 consensus_queue.cc:1035] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 8, Time since last communication: 0.000s
16:11:33.480 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:11:33.481 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)[L],3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843),d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)]}]
16:11:33.481 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=12, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=2097), Traces: [0ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [3ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [3ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [14ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [18ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [19ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [21ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [21ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [34ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [37ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [39ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [41ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [41ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [53ms] refreshing cache from master, [53ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [57ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [58ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [59ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [60ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [74ms] refreshing cache from master, [74ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [77ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [79ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [82ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [82ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [593ms] refreshing cache from master, [593ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [597ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [598ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [600ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [600ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [2093ms] refreshing cache from master, [2093ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [2097ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@1267402690(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@2443ab6 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1805664114 after retry RPC@1250009179) -> (continuation of Deferred@1760216081 after retry RPC@799945263) -> (continuation of Deferred@169222441 after retry RPC@1675867567) -> (continuation of Deferred@228127886 after retry RPC@1932031583) -> (continuation of Deferred@715651068 after retry RPC@1278891981), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4ba6c1b8 -> wakeup thread Time-limited test -> (continuation of Deferred@1805664114 after retry RPC@1250009179) -> (continuation of Deferred@1760216081 after retry RPC@799945263) -> (continuation of Deferred@169222441 after retry RPC@1675867567) -> (continuation of Deferred@228127886 after retry RPC@1932031583) -> (continuation of Deferred@715651068 after retry RPC@1278891981))) after lookup
16:11:33.482 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] connecting to peer
16:11:33.484 [DEBUG - kudu-nio-0] (Connection.java:757) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] Successfully connected to peer
16:11:33.507 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0x024448f3, L:/127.0.0.1:42696 - R:/127.16.234.130:43937] using TOKEN/null
16:11:33.569 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.16.234.129:42901
16:11:33.569 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

16:11:33.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:33.571127 17322 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu
16:11:33.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu
16:11:33.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-0/wal
16:11:33.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-0/data
16:11:33.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
16:11:33.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
16:11:33.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
16:11:33.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:11:33.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:11:33.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:11:33.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-0/logs
16:11:33.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-0/data/info.pb
16:11:33.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:11:33.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
16:11:33.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
16:11:33.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
16:11:33.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
16:11:33.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:11:33.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:11:33.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:11:33.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
16:11:33.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
16:11:33.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.234.129:42901
16:11:33.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.16.234.129
16:11:33.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.16.234.190:33901,127.16.234.189:32987,127.16.234.188:38065
16:11:33.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=43191
16:11:33.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.234.129
16:11:33.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.234.148:43519
16:11:33.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:11:33.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:11:33.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
16:11:33.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:33.879505 19349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
16:11:33.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:33.880030 19349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
16:11:33.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:33.880604 19349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
16:11:33.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:33.911653 19349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
16:11:33.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:33.912554 19349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.234.129
16:11:33.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:33.946503 19349 tablet_server_runner.cc:78] Tablet server non-default flags:
16:11:33.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:11:33.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.234.148:43519
16:11:33.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:11:33.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
16:11:33.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-0/data
16:11:33.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-0/wal
16:11:33.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:11:33.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:11:33.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.234.129:42901
16:11:33.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
16:11:33.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:11:33.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:11:33.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-0/data/info.pb
16:11:33.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.234.129
16:11:33.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=43191
16:11:33.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.16.234.190:33901,127.16.234.189:32987,127.16.234.188:38065
16:11:33.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
16:11:33.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:11:33.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
16:11:33.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
16:11:33.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:11:33.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.16.234.129
16:11:33.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-0/logs
16:11:33.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:11:33.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
16:11:33.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
16:11:33.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
16:11:33.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
16:11:33.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
16:11:33.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
16:11:33.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 15:12:52 UTC on ccycloud.quasar-tzrfpd.root.comops.site
16:11:33.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4580143
16:11:33.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
16:11:33.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:33.948086 19349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
16:11:33.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:33.949915 19349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
16:11:33.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:33.969456 19363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:11:35.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:33.969492 19367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:11:35.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:33.970917 19349 server_base.cc:1034] running on GCE node
16:11:35.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:33.969540 19364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:11:35.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:35.064826 19349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
16:11:35.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:35.067137 19349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
16:11:35.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:35.068496 19349 hybrid_clock.cc:648] HybridClock initialized: now 1786464695068466 us; error 48 us; skew 500 ppm
16:11:35.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:35.069283 19349 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
16:11:35.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:35.075425 19349 webserver.cc:460] Webserver started at http://127.16.234.129:43191/ using document root <none> and password file <none>
16:11:35.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:35.076296 19349 fs_manager.cc:359] Metadata directory not provided
16:11:35.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:35.076526 19349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
16:11:35.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:35.084393 19349 fs_manager.cc:709] Time spent opening directory manager: real 0.005s	user 0.005s	sys 0.000s
16:11:35.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:35.089016 19375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
16:11:35.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:35.090042 19349 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.002s	sys 0.000s
16:11:35.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:35.090359 19349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-0/data,/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-0/wal
16:11:35.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "d9315253ea9e464c9416773b52ee00a6"
16:11:35.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 16:11:14 on dist-test-slave-zj8q"
16:11:35.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:35.092634 19349 fs_report.cc:383] FS layout report
16:11:35.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
16:11:35.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-0/wal
16:11:35.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-0/wal
16:11:35.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-0/data/data
16:11:35.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
16:11:35.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
16:11:35.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
16:11:35.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
16:11:35.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
16:11:35.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
16:11:35.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
16:11:35.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
16:11:35.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
16:11:35.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
16:11:35.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
16:11:35.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:35.128985 19349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
16:11:35.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:35.130478 19349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
16:11:35.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:35.130923 19349 kserver.cc:163] Server-wide thread pool size limit: 3276
16:11:35.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:35.133342 19349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
16:11:35.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:35.140628 19385 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
16:11:35.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:35.153350 19349 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
16:11:35.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:35.153628 19349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.016s	user 0.000s	sys 0.003s
16:11:35.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:35.154052 19349 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
16:11:35.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:35.158951 19349 ts_tablet_manager.cc:610] Registered 1 tablets
16:11:35.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:35.159193 19349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s	user 0.001s	sys 0.003s
16:11:35.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:35.159550 19385 tablet_bootstrap.cc:492] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6: Bootstrap starting.
16:11:35.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:35.217599 19385 log.cc:824] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6: Log is configured to *not* fsync() on all Append() calls
16:11:35.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:35.263182 19385 tablet_bootstrap.cc:492] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=9 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
16:11:35.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:35.264043 19385 tablet_bootstrap.cc:492] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6: Bootstrap complete.
16:11:35.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:35.265249 19385 ts_tablet_manager.cc:1397] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6: Time spent bootstrapping tablet: real 0.106s	user 0.073s	sys 0.027s
16:11:35.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:35.289255 19385 raft_consensus.cc:348] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:11:35.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:35.292829 19385 raft_consensus.cc:729] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9315253ea9e464c9416773b52ee00a6, State: Initialized, Role: FOLLOWER
16:11:35.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:35.293911 19385 consensus_queue.cc:260] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 4.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:11:35.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:35.298877 19385 ts_tablet_manager.cc:1428] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6: Time spent starting tablet: real 0.033s	user 0.030s	sys 0.005s
16:11:35.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:35.333509 19349 rpc_server.cc:307] RPC server started. Bound to: 127.16.234.129:42901
16:11:35.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:35.334025 19566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.234.129:42901 every 8 connection(s)
16:11:35.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:35.336097 19349 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-0/data/info.pb
16:11:35.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:35.343792 17322 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu as pid 19349
16:11:35.344 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
16:11:35.372 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 12 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304, tablet=7d3c128d101444a59e009013a5c634dc, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
16:11:35.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:35.381011 19567 heartbeater.cc:347] Connected to a master server at 127.16.234.188:38065
16:11:35.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:35.381523 19567 heartbeater.cc:464] Registering TS with master...
16:11:35.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:35.382733 19567 heartbeater.cc:511] Master 127.16.234.188:38065 requested a full tablet report, sending...
16:11:35.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:35.385609 19569 heartbeater.cc:347] Connected to a master server at 127.16.234.190:33901
16:11:35.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:35.385953 19569 heartbeater.cc:464] Registering TS with master...
16:11:35.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:35.386780 19569 heartbeater.cc:511] Master 127.16.234.190:33901 requested a full tablet report, sending...
16:11:35.387 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.16.234.130:43937
16:11:35.388 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

16:11:35.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:35.388446 17322 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu with pid 18753
16:11:35.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:35.396611 17401 ts_manager.cc:194] Re-registered known tserver with Master: d9315253ea9e464c9416773b52ee00a6 (127.16.234.129:42901)
16:11:35.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:35.398034 19568 heartbeater.cc:347] Connected to a master server at 127.16.234.189:32987
16:11:35.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:35.398411 19568 heartbeater.cc:464] Registering TS with master...
16:11:35.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:35.399226 19568 heartbeater.cc:511] Master 127.16.234.189:32987 requested a full tablet report, sending...
16:11:35.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:35.402320 17659 ts_manager.cc:194] Re-registered known tserver with Master: d9315253ea9e464c9416773b52ee00a6 (127.16.234.129:42901)
16:11:35.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:35.403913 17536 ts_manager.cc:194] Re-registered known tserver with Master: d9315253ea9e464c9416773b52ee00a6 (127.16.234.129:42901)
16:11:35.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:35.405815 17659 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.234.129:35257
16:11:35.429 [DEBUG - kudu-nio-0] (Connection.java:239) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] handling channelInactive
16:11:35.429 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] cleaning up while in state READY due to: connection closed
16:11:35.430 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
16:11:35.431 [DEBUG - Time-limited test] (Connection.java:748) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] connecting to peer
16:11:35.432 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:11:35.432 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:11:35.432 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:11:35.440 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:11:35.440 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)[L],3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843),d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)]}]
16:11:35.440 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=2, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=10), Traces: [0ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [2ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [2ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [9ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@288953988(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@4c1629e0 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6c9a1901 -> wakeup thread Time-limited test)) after lookup
16:11:35.441 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] connecting to peer
16:11:35.443 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:11:35.443 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:11:35.443 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:11:35.460 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:11:35.460 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)[L],3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843),d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)]}]
16:11:35.460 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=4, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=30), Traces: [0ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [2ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [2ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [9ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [11ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [13ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [13ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [26ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [30ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@288953988(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@4c1629e0 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@600948190 after retry RPC@524282548), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6c9a1901 -> wakeup thread Time-limited test -> (continuation of Deferred@600948190 after retry RPC@524282548))) after lookup
16:11:35.461 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] connecting to peer
16:11:35.462 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:11:35.462 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:11:35.462 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:11:35.501 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:11:35.501 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)[L],3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843),d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)]}]
16:11:35.502 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=6, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=71), Traces: [0ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [2ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [2ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [9ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [11ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [13ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [13ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [26ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [30ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [31ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [32ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [32ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [66ms] refreshing cache from master, [67ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [71ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@288953988(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@4c1629e0 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@600948190 after retry RPC@524282548) -> (continuation of Deferred@1337991636 after retry RPC@1207067707), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6c9a1901 -> wakeup thread Time-limited test -> (continuation of Deferred@600948190 after retry RPC@524282548) -> (continuation of Deferred@1337991636 after retry RPC@1207067707))) after lookup
16:11:35.503 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] connecting to peer
16:11:35.504 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:11:35.505 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:11:35.505 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:11:35.560 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:11:35.560 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)[L],3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843),d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)]}]
16:11:35.561 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=8, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=130), Traces: [0ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [2ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [2ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [9ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [11ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [13ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [13ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [26ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [30ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [31ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [32ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [32ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [66ms] refreshing cache from master, [67ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [71ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [72ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [75ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [75ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [126ms] refreshing cache from master, [126ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [130ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@288953988(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@4c1629e0 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@600948190 after retry RPC@524282548) -> (continuation of Deferred@1337991636 after retry RPC@1207067707) -> (continuation of Deferred@939745604 after retry RPC@1812203971), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6c9a1901 -> wakeup thread Time-limited test -> (continuation of Deferred@600948190 after retry RPC@524282548) -> (continuation of Deferred@1337991636 after retry RPC@1207067707) -> (continuation of Deferred@939745604 after retry RPC@1812203971))) after lookup
16:11:35.562 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] connecting to peer
16:11:35.562 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:11:35.563 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:11:35.563 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:11:35.641 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:11:35.641 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)[L],3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843),d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)]}]
16:11:35.641 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=10, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=210), Traces: [0ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [2ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [2ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [9ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [11ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [13ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [13ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [26ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [30ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [31ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [32ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [32ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [66ms] refreshing cache from master, [67ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [71ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [72ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [75ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [75ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [126ms] refreshing cache from master, [126ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [130ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [131ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [133ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [133ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [207ms] refreshing cache from master, [207ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [210ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@288953988(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@4c1629e0 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@600948190 after retry RPC@524282548) -> (continuation of Deferred@1337991636 after retry RPC@1207067707) -> (continuation of Deferred@939745604 after retry RPC@1812203971) -> (continuation of Deferred@1106948132 after retry RPC@1571151181), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6c9a1901 -> wakeup thread Time-limited test -> (continuation of Deferred@600948190 after retry RPC@524282548) -> (continuation of Deferred@1337991636 after retry RPC@1207067707) -> (continuation of Deferred@939745604 after retry RPC@1812203971) -> (continuation of Deferred@1106948132 after retry RPC@1571151181))) after lookup
16:11:35.642 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] connecting to peer
16:11:35.644 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:11:35.644 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:11:35.644 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:11:36.100 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:11:36.101 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)[L],3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843),d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)]}]
16:11:36.102 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=12, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=671), Traces: [0ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [2ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [2ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [9ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [11ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [13ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [13ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [26ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [30ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [31ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [32ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [32ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [66ms] refreshing cache from master, [67ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [71ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [72ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [75ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [75ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [126ms] refreshing cache from master, [126ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [130ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [131ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [133ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [133ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [207ms] refreshing cache from master, [207ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [210ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [212ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [214ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [214ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [666ms] refreshing cache from master, [666ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [670ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@288953988(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@4c1629e0 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@600948190 after retry RPC@524282548) -> (continuation of Deferred@1337991636 after retry RPC@1207067707) -> (continuation of Deferred@939745604 after retry RPC@1812203971) -> (continuation of Deferred@1106948132 after retry RPC@1571151181) -> (continuation of Deferred@1430006852 after retry RPC@645979163), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6c9a1901 -> wakeup thread Time-limited test -> (continuation of Deferred@600948190 after retry RPC@524282548) -> (continuation of Deferred@1337991636 after retry RPC@1207067707) -> (continuation of Deferred@939745604 after retry RPC@1812203971) -> (continuation of Deferred@1106948132 after retry RPC@1571151181) -> (continuation of Deferred@1430006852 after retry RPC@645979163))) after lookup
16:11:36.102 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] connecting to peer
16:11:36.103 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:11:36.103 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:11:36.103 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:11:36.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:36.409979 19567 heartbeater.cc:503] Master 127.16.234.188:38065 was elected leader, sending a full tablet report...
16:11:36.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:36.515022 19582 raft_consensus.cc:482] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
16:11:36.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:36.515587 19582 raft_consensus.cc:504] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:11:36.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:36.517719 19582 leader_election.cc:288] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d392ba816c24499a0f7fecd95f88c15 (127.16.234.131:42843), 1bce670edc6c40b2bb1584ec3b364e0f (127.16.234.130:43937)
16:11:36.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:36.521221 19382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.234.130:43937: connect: Connection refused (error 111)
16:11:36.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:36.524883 19382 leader_election.cc:334] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 1bce670edc6c40b2bb1584ec3b364e0f (127.16.234.130:43937): Network error: Client connection negotiation failed: client connection to 127.16.234.130:43937: connect: Connection refused (error 111)
16:11:36.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:36.529419 19202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d3c128d101444a59e009013a5c634dc" candidate_uuid: "d9315253ea9e464c9416773b52ee00a6" candidate_term: 5 candidate_status { last_received { term: 4 index: 12 } } ignore_live_leader: false dest_uuid: "3d392ba816c24499a0f7fecd95f88c15" is_pre_election: true
16:11:36.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:36.530841 19378 leader_election.cc:302] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9315253ea9e464c9416773b52ee00a6; no voters: 1bce670edc6c40b2bb1584ec3b364e0f, 3d392ba816c24499a0f7fecd95f88c15
16:11:36.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:36.531471 19582 raft_consensus.cc:2726] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
16:11:36.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:36.755873 19592 raft_consensus.cc:482] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1bce670edc6c40b2bb1584ec3b364e0f)
16:11:36.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:36.756318 19592 raft_consensus.cc:504] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:11:36.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:36.758752 19592 leader_election.cc:288] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d9315253ea9e464c9416773b52ee00a6 (127.16.234.129:42901), 1bce670edc6c40b2bb1584ec3b364e0f (127.16.234.130:43937)
16:11:36.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:36.762836 19105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.234.130:43937: connect: Connection refused (error 111)
16:11:36.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:36.766908 19105 leader_election.cc:334] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 1bce670edc6c40b2bb1584ec3b364e0f (127.16.234.130:43937): Network error: Client connection negotiation failed: client connection to 127.16.234.130:43937: connect: Connection refused (error 111)
16:11:36.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:36.773005 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d3c128d101444a59e009013a5c634dc" candidate_uuid: "3d392ba816c24499a0f7fecd95f88c15" candidate_term: 6 candidate_status { last_received { term: 5 index: 16 } } ignore_live_leader: false dest_uuid: "d9315253ea9e464c9416773b52ee00a6" is_pre_election: true
16:11:36.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:36.773662 19489 raft_consensus.cc:2454] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d392ba816c24499a0f7fecd95f88c15 in term 4.
16:11:36.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:36.774920 19103 leader_election.cc:302] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d392ba816c24499a0f7fecd95f88c15, d9315253ea9e464c9416773b52ee00a6; no voters: 1bce670edc6c40b2bb1584ec3b364e0f
16:11:36.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:36.775609 19592 raft_consensus.cc:2781] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 5 FOLLOWER]: Leader pre-election won for term 6
16:11:36.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:36.775894 19592 raft_consensus.cc:482] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1bce670edc6c40b2bb1584ec3b364e0f)
16:11:36.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:36.776118 19592 raft_consensus.cc:3037] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 5 FOLLOWER]: Advancing to term 6
16:11:36.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:36.780427 19592 raft_consensus.cc:504] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:11:36.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:36.781872 19592 leader_election.cc:288] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [CANDIDATE]: Term 6 election: Requested vote from peers d9315253ea9e464c9416773b52ee00a6 (127.16.234.129:42901), 1bce670edc6c40b2bb1584ec3b364e0f (127.16.234.130:43937)
16:11:36.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:36.782632 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d3c128d101444a59e009013a5c634dc" candidate_uuid: "3d392ba816c24499a0f7fecd95f88c15" candidate_term: 6 candidate_status { last_received { term: 5 index: 16 } } ignore_live_leader: false dest_uuid: "d9315253ea9e464c9416773b52ee00a6"
16:11:36.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:36.783211 19489 raft_consensus.cc:3037] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 4 FOLLOWER]: Advancing to term 6
16:11:36.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:36.785063 19105 leader_election.cc:334] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 1bce670edc6c40b2bb1584ec3b364e0f (127.16.234.130:43937): Network error: Client connection negotiation failed: client connection to 127.16.234.130:43937: connect: Connection refused (error 111)
16:11:36.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:36.789987 19489 raft_consensus.cc:2454] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d392ba816c24499a0f7fecd95f88c15 in term 6.
16:11:36.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:36.790917 19103 leader_election.cc:302] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d392ba816c24499a0f7fecd95f88c15, d9315253ea9e464c9416773b52ee00a6; no voters: 1bce670edc6c40b2bb1584ec3b364e0f
16:11:36.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:36.791524 19592 raft_consensus.cc:2781] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 6 FOLLOWER]: Leader election won for term 6
16:11:36.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:36.793032 19592 raft_consensus.cc:686] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 6 LEADER]: Becoming Leader. State: Replica: 3d392ba816c24499a0f7fecd95f88c15, State: Running, Role: LEADER
16:11:36.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:36.793776 19592 consensus_queue.cc:237] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 5.16, Last appended by leader: 16, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:11:36.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:36.803295 17659 catalog_manager.cc:5495] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 reported cstate change: term changed from 5 to 6, leader changed from 1bce670edc6c40b2bb1584ec3b364e0f (127.16.234.130) to 3d392ba816c24499a0f7fecd95f88c15 (127.16.234.131). New cstate: current_term: 6 leader_uuid: "3d392ba816c24499a0f7fecd95f88c15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } health_report { overall_health: UNKNOWN } } }
16:11:36.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:36.809116 17703 debug-util.cc:398] Leaking SignalData structure 0x7b08000b0fc0 after lost signal to thread 17609
16:11:36.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:36.809752 17703 debug-util.cc:398] Leaking SignalData structure 0x7b08000b9a20 after lost signal to thread 17710
16:11:37.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:37.257112 19105 consensus_peers.cc:487] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 -> Peer 1bce670edc6c40b2bb1584ec3b364e0f (127.16.234.130:43937): Couldn't send request to peer 1bce670edc6c40b2bb1584ec3b364e0f. Status: Network error: Client connection negotiation failed: client connection to 127.16.234.130:43937: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
16:11:37.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:37.387966 19489 raft_consensus.cc:1261] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 6 FOLLOWER]: Refusing update from remote peer 3d392ba816c24499a0f7fecd95f88c15: Log matching property violated. Preceding OpId in replica: term: 4 index: 12. Preceding OpId from leader: term: 6 index: 17. (index mismatch)
16:11:37.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:37.389395 19592 consensus_queue.cc:1035] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 12, Time since last communication: 0.000s
16:11:39.560 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:11:39.561 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937),3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)[L],d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)]}]
16:11:39.561 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=14, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=4130), Traces: [0ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [2ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [2ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [9ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [11ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [13ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [13ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [26ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [30ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [31ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [32ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [32ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [66ms] refreshing cache from master, [67ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [71ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [72ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [75ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [75ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [126ms] refreshing cache from master, [126ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [130ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [131ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [133ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [133ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [207ms] refreshing cache from master, [207ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [210ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [212ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [214ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [214ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [666ms] refreshing cache from master, [666ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [670ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [672ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [673ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [673ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [4126ms] refreshing cache from master, [4126ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [4130ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@288953988(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@4c1629e0 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@600948190 after retry RPC@524282548) -> (continuation of Deferred@1337991636 after retry RPC@1207067707) -> (continuation of Deferred@939745604 after retry RPC@1812203971) -> (continuation of Deferred@1106948132 after retry RPC@1571151181) -> (continuation of Deferred@1430006852 after retry RPC@645979163) -> (continuation of Deferred@1627212612 after retry RPC@241996669), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6c9a1901 -> wakeup thread Time-limited test -> (continuation of Deferred@600948190 after retry RPC@524282548) -> (continuation of Deferred@1337991636 after retry RPC@1207067707) -> (continuation of Deferred@939745604 after retry RPC@1812203971) -> (continuation of Deferred@1106948132 after retry RPC@1571151181) -> (continuation of Deferred@1430006852 after retry RPC@645979163) -> (continuation of Deferred@1627212612 after retry RPC@241996669))) after lookup
16:11:39.562 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)] connecting to peer
16:11:39.564 [DEBUG - kudu-nio-0] (Connection.java:757) [peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)] Successfully connected to peer
16:11:39.585 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0x4e4ade85, L:/127.0.0.1:34388 - R:/127.16.234.131:42843] using TOKEN/null
16:11:39.630 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.16.234.130:43937
16:11:39.630 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

16:11:39.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:39.632237 17322 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu
16:11:39.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu
16:11:39.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/wal
16:11:39.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/data
16:11:39.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
16:11:39.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
16:11:39.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
16:11:39.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:11:39.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:11:39.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:11:39.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/logs
16:11:39.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/data/info.pb
16:11:39.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:11:39.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
16:11:39.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
16:11:39.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
16:11:39.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
16:11:39.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:11:39.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:11:39.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:11:39.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
16:11:39.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
16:11:39.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.234.130:43937
16:11:39.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.16.234.130
16:11:39.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.16.234.190:33901,127.16.234.189:32987,127.16.234.188:38065
16:11:39.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=37863
16:11:39.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.234.130
16:11:39.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.234.148:43519
16:11:39.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:11:39.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:11:39.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
16:11:39.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:39.727231 19105 consensus_peers.cc:487] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 -> Peer 1bce670edc6c40b2bb1584ec3b364e0f (127.16.234.130:43937): Couldn't send request to peer 1bce670edc6c40b2bb1584ec3b364e0f. Status: Network error: Client connection negotiation failed: client connection to 127.16.234.130:43937: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
16:11:39.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:39.941927 19655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
16:11:39.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:39.942487 19655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
16:11:39.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:39.943132 19655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
16:11:39.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:39.974968 19655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
16:11:39.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:39.975889 19655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.234.130
16:11:40.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:40.011113 19655 tablet_server_runner.cc:78] Tablet server non-default flags:
16:11:40.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:11:40.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.234.148:43519
16:11:40.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:11:40.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
16:11:40.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/data
16:11:40.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/wal
16:11:40.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:11:40.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:11:40.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.234.130:43937
16:11:40.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
16:11:40.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:11:40.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:11:40.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/data/info.pb
16:11:40.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.234.130
16:11:40.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=37863
16:11:40.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.16.234.190:33901,127.16.234.189:32987,127.16.234.188:38065
16:11:40.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
16:11:40.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:11:40.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
16:11:40.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
16:11:40.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:11:40.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.16.234.130
16:11:40.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/logs
16:11:40.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:11:40.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
16:11:40.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
16:11:40.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
16:11:40.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
16:11:40.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
16:11:40.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
16:11:40.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 15:12:52 UTC on ccycloud.quasar-tzrfpd.root.comops.site
16:11:40.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4580143
16:11:40.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
16:11:40.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:40.012459 19655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
16:11:40.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:40.014696 19655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
16:11:40.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:40.035041 19669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:11:41.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:40.035755 19668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:11:41.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:40.036321 19655 server_base.cc:1034] running on GCE node
16:11:41.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:40.035209 19671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:11:41.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:41.144913 19655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
16:11:41.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:41.147874 19655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
16:11:41.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:41.149338 19655 hybrid_clock.cc:648] HybridClock initialized: now 1786464701149290 us; error 51 us; skew 500 ppm
16:11:41.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:41.150405 19655 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
16:11:41.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:41.163175 19655 webserver.cc:460] Webserver started at http://127.16.234.130:37863/ using document root <none> and password file <none>
16:11:41.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:41.164377 19655 fs_manager.cc:359] Metadata directory not provided
16:11:41.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:41.164650 19655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
16:11:41.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:41.175161 19655 fs_manager.cc:709] Time spent opening directory manager: real 0.006s	user 0.007s	sys 0.001s
16:11:41.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:41.180027 19687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
16:11:41.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:41.181258 19655 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.001s	sys 0.004s
16:11:41.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:41.181562 19655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/data,/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/wal
16:11:41.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "1bce670edc6c40b2bb1584ec3b364e0f"
16:11:41.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 16:11:16 on dist-test-slave-zj8q"
16:11:41.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:41.183957 19655 fs_report.cc:383] FS layout report
16:11:41.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
16:11:41.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/wal
16:11:41.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/wal
16:11:41.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/data/data
16:11:41.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
16:11:41.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
16:11:41.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
16:11:41.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
16:11:41.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
16:11:41.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
16:11:41.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
16:11:41.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
16:11:41.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
16:11:41.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
16:11:41.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
16:11:41.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:41.228633 19655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
16:11:41.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:41.230108 19655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
16:11:41.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:41.230548 19655 kserver.cc:163] Server-wide thread pool size limit: 3276
16:11:41.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:41.233037 19655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
16:11:41.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:41.238782 19697 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
16:11:41.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:41.247582 19655 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
16:11:41.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:41.247810 19655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.011s	user 0.000s	sys 0.002s
16:11:41.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:41.248020 19655 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
16:11:41.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:41.252775 19655 ts_tablet_manager.cc:610] Registered 1 tablets
16:11:41.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:41.252952 19655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s	user 0.001s	sys 0.001s
16:11:41.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:41.253484 19697 tablet_bootstrap.cc:492] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f: Bootstrap starting.
16:11:41.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:41.310921 19697 log.cc:824] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f: Log is configured to *not* fsync() on all Append() calls
16:11:41.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:41.380880 19697 tablet_bootstrap.cc:492] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f: Bootstrap replayed 1/1 log segments. Stats: ops{read=16 overwritten=0 applied=16 ignored=0} inserts{seen=12 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
16:11:41.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:41.381995 19697 tablet_bootstrap.cc:492] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f: Bootstrap complete.
16:11:41.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:41.383888 19697 ts_tablet_manager.cc:1397] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f: Time spent bootstrapping tablet: real 0.131s	user 0.094s	sys 0.027s
16:11:41.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:41.410360 19697 raft_consensus.cc:348] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:11:41.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:41.413923 19697 raft_consensus.cc:729] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bce670edc6c40b2bb1584ec3b364e0f, State: Initialized, Role: FOLLOWER
16:11:41.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:41.415302 19697 consensus_queue.cc:260] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 5.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:11:41.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:41.419616 19697 ts_tablet_manager.cc:1428] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f: Time spent starting tablet: real 0.035s	user 0.028s	sys 0.008s
16:11:41.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:41.441921 19655 rpc_server.cc:307] RPC server started. Bound to: 127.16.234.130:43937
16:11:41.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:41.442097 19878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.234.130:43937 every 8 connection(s)
16:11:41.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:41.445060 19655 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/data/info.pb
16:11:41.452 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
16:11:41.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:41.449326 17322 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu as pid 19655
16:11:41.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:41.481520 19805 raft_consensus.cc:3037] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [term 5 FOLLOWER]: Advancing to term 6
16:11:41.483 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 15 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304, tablet=7d3c128d101444a59e009013a5c634dc, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
16:11:41.484 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.16.234.131:42843
16:11:41.485 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

16:11:41.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:41.486001 17322 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu with pid 19055
16:11:41.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:41.545271 19879 heartbeater.cc:347] Connected to a master server at 127.16.234.188:38065
16:11:41.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:41.545849 19879 heartbeater.cc:464] Registering TS with master...
16:11:41.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:41.547326 19879 heartbeater.cc:511] Master 127.16.234.188:38065 requested a full tablet report, sending...
16:11:41.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:41.553308 19882 heartbeater.cc:347] Connected to a master server at 127.16.234.189:32987
16:11:41.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:41.553690 19882 heartbeater.cc:464] Registering TS with master...
16:11:41.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:41.554618 19882 heartbeater.cc:511] Master 127.16.234.189:32987 requested a full tablet report, sending...
16:11:41.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:41.559669 19689 connection.cc:535] server connection from 127.16.234.131:60811 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
16:11:41.560 [DEBUG - kudu-nio-0] (Connection.java:239) [peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)] handling channelInactive
16:11:41.561 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)] cleaning up while in state READY due to: connection closed
16:11:41.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:41.564890 17536 ts_manager.cc:194] Re-registered known tserver with Master: 1bce670edc6c40b2bb1584ec3b364e0f (127.16.234.130:43937)
16:11:41.568 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
16:11:41.569 [DEBUG - Time-limited test] (Connection.java:748) [peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)] connecting to peer
16:11:41.571 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843
16:11:41.571 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843
16:11:41.571 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843
16:11:41.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:41.569448 17662 ts_manager.cc:194] Re-registered known tserver with Master: 1bce670edc6c40b2bb1584ec3b364e0f (127.16.234.130:43937)
16:11:41.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:41.572777 17662 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.234.130:44151
16:11:41.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:41.576170 19879 heartbeater.cc:503] Master 127.16.234.188:38065 was elected leader, sending a full tablet report...
16:11:41.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:41.578548 19893 heartbeater.cc:347] Connected to a master server at 127.16.234.190:33901
16:11:41.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:41.578907 19893 heartbeater.cc:464] Registering TS with master...
16:11:41.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:41.579581 19893 heartbeater.cc:511] Master 127.16.234.190:33901 requested a full tablet report, sending...
16:11:41.582 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:11:41.582 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937),3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)[L],d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)]}]
16:11:41.582 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=2, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=14), Traces: [0ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [3ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [3ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [13ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@2011673307(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@22aa2ecd -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@13e2208d -> wakeup thread Time-limited test)) after lookup
16:11:41.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:41.582648 17401 ts_manager.cc:194] Re-registered known tserver with Master: 1bce670edc6c40b2bb1584ec3b364e0f (127.16.234.130:43937)
16:11:41.583 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)] connecting to peer
16:11:41.584 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843
16:11:41.585 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843
16:11:41.585 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843
16:11:41.600 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:11:41.601 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937),3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)[L],d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)]}]
16:11:41.601 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=4, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=33), Traces: [0ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [3ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [3ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [13ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [15ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [17ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [17ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [32ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@2011673307(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@22aa2ecd -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@501359163 after retry RPC@528916290), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@13e2208d -> wakeup thread Time-limited test -> (continuation of Deferred@501359163 after retry RPC@528916290))) after lookup
16:11:41.602 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)] connecting to peer
16:11:41.603 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843
16:11:41.603 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843
16:11:41.603 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843
16:11:41.620 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:11:41.621 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937),3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)[L],d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)]}]
16:11:41.621 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=6, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=52), Traces: [0ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [3ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [3ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [13ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [15ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [17ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [17ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [32ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [33ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [35ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [35ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [48ms] refreshing cache from master, [48ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [52ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@2011673307(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@22aa2ecd -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@501359163 after retry RPC@528916290) -> (continuation of Deferred@961393740 after retry RPC@98987009), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@13e2208d -> wakeup thread Time-limited test -> (continuation of Deferred@501359163 after retry RPC@528916290) -> (continuation of Deferred@961393740 after retry RPC@98987009))) after lookup
16:11:41.621 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)] connecting to peer
16:11:41.623 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843
16:11:41.623 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843
16:11:41.623 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843
16:11:41.660 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:11:41.661 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937),3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)[L],d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)]}]
16:11:41.661 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=8, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=93), Traces: [0ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [3ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [3ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [13ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [15ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [17ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [17ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [32ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [33ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [35ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [35ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [48ms] refreshing cache from master, [48ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [52ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [53ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [55ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [55ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [88ms] refreshing cache from master, [88ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [92ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@2011673307(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@22aa2ecd -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@501359163 after retry RPC@528916290) -> (continuation of Deferred@961393740 after retry RPC@98987009) -> (continuation of Deferred@492297528 after retry RPC@1693593363), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@13e2208d -> wakeup thread Time-limited test -> (continuation of Deferred@501359163 after retry RPC@528916290) -> (continuation of Deferred@961393740 after retry RPC@98987009) -> (continuation of Deferred@492297528 after retry RPC@1693593363))) after lookup
16:11:41.662 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)] connecting to peer
16:11:41.663 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843
16:11:41.663 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843
16:11:41.663 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843
16:11:41.941 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:11:41.941 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937),3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)[L],d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)]}]
16:11:41.941 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=10, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=373), Traces: [0ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [3ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [3ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [13ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [15ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [17ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [17ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [32ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [33ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [35ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [35ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [48ms] refreshing cache from master, [48ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [52ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [53ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [55ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [55ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [88ms] refreshing cache from master, [88ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [92ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [93ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [95ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [95ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [368ms] refreshing cache from master, [369ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [372ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@2011673307(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@22aa2ecd -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@501359163 after retry RPC@528916290) -> (continuation of Deferred@961393740 after retry RPC@98987009) -> (continuation of Deferred@492297528 after retry RPC@1693593363) -> (continuation of Deferred@1501197081 after retry RPC@652955106), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@13e2208d -> wakeup thread Time-limited test -> (continuation of Deferred@501359163 after retry RPC@528916290) -> (continuation of Deferred@961393740 after retry RPC@98987009) -> (continuation of Deferred@492297528 after retry RPC@1693593363) -> (continuation of Deferred@1501197081 after retry RPC@652955106))) after lookup
16:11:41.942 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)] connecting to peer
16:11:41.944 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843
16:11:41.944 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843
16:11:41.944 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843
16:11:42.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:42.567178 19899 raft_consensus.cc:482] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3d392ba816c24499a0f7fecd95f88c15)
16:11:42.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:42.567567 19899 raft_consensus.cc:504] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:11:42.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:42.569003 19899 leader_election.cc:288] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d392ba816c24499a0f7fecd95f88c15 (127.16.234.131:42843), 1bce670edc6c40b2bb1584ec3b364e0f (127.16.234.130:43937)
16:11:42.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:42.579413 19378 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.234.131:42843: connect: Connection refused (error 111)
16:11:42.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:42.582412 19378 leader_election.cc:334] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 3d392ba816c24499a0f7fecd95f88c15 (127.16.234.131:42843): Network error: Client connection negotiation failed: client connection to 127.16.234.131:42843: connect: Connection refused (error 111)
16:11:42.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:42.589663 19805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d3c128d101444a59e009013a5c634dc" candidate_uuid: "d9315253ea9e464c9416773b52ee00a6" candidate_term: 7 candidate_status { last_received { term: 6 index: 20 } } ignore_live_leader: false dest_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" is_pre_election: true
16:11:42.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:42.591043 19382 leader_election.cc:302] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9315253ea9e464c9416773b52ee00a6; no voters: 1bce670edc6c40b2bb1584ec3b364e0f, 3d392ba816c24499a0f7fecd95f88c15
16:11:42.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:42.591671 19899 raft_consensus.cc:2726] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
16:11:42.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:42.997069 19908 raft_consensus.cc:482] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3d392ba816c24499a0f7fecd95f88c15)
16:11:42.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:42.997565 19908 raft_consensus.cc:504] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:11:43.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:42.999790 19908 leader_election.cc:288] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d392ba816c24499a0f7fecd95f88c15 (127.16.234.131:42843), d9315253ea9e464c9416773b52ee00a6 (127.16.234.129:42901)
16:11:43.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:43.003281 19689 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.234.131:42843: connect: Connection refused (error 111)
16:11:43.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:43.007925 19689 leader_election.cc:334] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 3d392ba816c24499a0f7fecd95f88c15 (127.16.234.131:42843): Network error: Client connection negotiation failed: client connection to 127.16.234.131:42843: connect: Connection refused (error 111)
16:11:43.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:43.011504 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d3c128d101444a59e009013a5c634dc" candidate_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" candidate_term: 7 candidate_status { last_received { term: 6 index: 20 } } ignore_live_leader: false dest_uuid: "d9315253ea9e464c9416773b52ee00a6" is_pre_election: true
16:11:43.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:43.012006 19489 raft_consensus.cc:2454] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bce670edc6c40b2bb1584ec3b364e0f in term 6.
16:11:43.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:43.013044 19690 leader_election.cc:302] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1bce670edc6c40b2bb1584ec3b364e0f, d9315253ea9e464c9416773b52ee00a6; no voters: 3d392ba816c24499a0f7fecd95f88c15
16:11:43.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:43.013763 19908 raft_consensus.cc:2781] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [term 6 FOLLOWER]: Leader pre-election won for term 7
16:11:43.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:43.014014 19908 raft_consensus.cc:482] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [term 6 FOLLOWER]: Starting leader election (detected failure of leader 3d392ba816c24499a0f7fecd95f88c15)
16:11:43.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:43.014317 19908 raft_consensus.cc:3037] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [term 6 FOLLOWER]: Advancing to term 7
16:11:43.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:43.018462 19908 raft_consensus.cc:504] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:11:43.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:43.019807 19908 leader_election.cc:288] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [CANDIDATE]: Term 7 election: Requested vote from peers 3d392ba816c24499a0f7fecd95f88c15 (127.16.234.131:42843), d9315253ea9e464c9416773b52ee00a6 (127.16.234.129:42901)
16:11:43.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:43.020762 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d3c128d101444a59e009013a5c634dc" candidate_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" candidate_term: 7 candidate_status { last_received { term: 6 index: 20 } } ignore_live_leader: false dest_uuid: "d9315253ea9e464c9416773b52ee00a6"
16:11:43.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:43.021200 19489 raft_consensus.cc:3037] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 6 FOLLOWER]: Advancing to term 7
16:11:43.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:43.022620 19689 leader_election.cc:334] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 3d392ba816c24499a0f7fecd95f88c15 (127.16.234.131:42843): Network error: Client connection negotiation failed: client connection to 127.16.234.131:42843: connect: Connection refused (error 111)
16:11:43.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:43.025246 19489 raft_consensus.cc:2454] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bce670edc6c40b2bb1584ec3b364e0f in term 7.
16:11:43.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:43.026027 19690 leader_election.cc:302] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1bce670edc6c40b2bb1584ec3b364e0f, d9315253ea9e464c9416773b52ee00a6; no voters: 3d392ba816c24499a0f7fecd95f88c15
16:11:43.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:43.026652 19908 raft_consensus.cc:2781] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [term 7 FOLLOWER]: Leader election won for term 7
16:11:43.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:43.028051 19908 raft_consensus.cc:686] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [term 7 LEADER]: Becoming Leader. State: Replica: 1bce670edc6c40b2bb1584ec3b364e0f, State: Running, Role: LEADER
16:11:43.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:43.028904 19908 consensus_queue.cc:237] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 6.20, Last appended by leader: 20, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:11:43.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:43.037544 17659 catalog_manager.cc:5495] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f reported cstate change: term changed from 6 to 7, leader changed from 3d392ba816c24499a0f7fecd95f88c15 (127.16.234.131) to 1bce670edc6c40b2bb1584ec3b364e0f (127.16.234.130). New cstate: current_term: 7 leader_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } health_report { overall_health: HEALTHY } } }
16:11:43.360 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:11:43.361 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)[L],3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843),d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)]}]
16:11:43.361 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=12, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=1792), Traces: [0ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [3ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [3ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [13ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [15ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [17ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [17ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [32ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [33ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [35ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [35ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [48ms] refreshing cache from master, [48ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [52ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [53ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [55ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [55ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [88ms] refreshing cache from master, [88ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [92ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [93ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [95ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [95ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [368ms] refreshing cache from master, [369ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [372ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [374ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [376ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [376ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [1788ms] refreshing cache from master, [1789ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [1792ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@2011673307(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@22aa2ecd -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@501359163 after retry RPC@528916290) -> (continuation of Deferred@961393740 after retry RPC@98987009) -> (continuation of Deferred@492297528 after retry RPC@1693593363) -> (continuation of Deferred@1501197081 after retry RPC@652955106) -> (continuation of Deferred@550511684 after retry RPC@1554577889), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@13e2208d -> wakeup thread Time-limited test -> (continuation of Deferred@501359163 after retry RPC@528916290) -> (continuation of Deferred@961393740 after retry RPC@98987009) -> (continuation of Deferred@492297528 after retry RPC@1693593363) -> (continuation of Deferred@1501197081 after retry RPC@652955106) -> (continuation of Deferred@550511684 after retry RPC@1554577889))) after lookup
16:11:43.362 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] connecting to peer
16:11:43.363 [DEBUG - kudu-nio-1] (Connection.java:757) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] Successfully connected to peer
16:11:43.409 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x6a92149e, L:/127.0.0.1:50348 - R:/127.16.234.130:43937] using TOKEN/null
16:11:43.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:43.419577 19489 raft_consensus.cc:1261] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 7 FOLLOWER]: Refusing update from remote peer 1bce670edc6c40b2bb1584ec3b364e0f: Log matching property violated. Preceding OpId in replica: term: 6 index: 20. Preceding OpId from leader: term: 7 index: 22. (index mismatch)
16:11:43.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:43.421130 19922 consensus_queue.cc:1035] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
16:11:43.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:43.421777 19689 consensus_peers.cc:487] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f -> Peer 3d392ba816c24499a0f7fecd95f88c15 (127.16.234.131:42843): Couldn't send request to peer 3d392ba816c24499a0f7fecd95f88c15. Status: Network error: Client connection negotiation failed: client connection to 127.16.234.131:42843: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
16:11:43.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:43.432384 19944 mvcc.cc:205] Tried to move back new op lower bound from 7317359425190531072 to 7317359423613739008. Current Snapshot: MvccSnapshot[applied={T|T < 7317359425190531072}]
16:11:43.479 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.16.234.131:42843
16:11:43.481 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

16:11:43.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:43.482687 17322 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu
16:11:43.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu
16:11:43.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-2/wal
16:11:43.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-2/data
16:11:43.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
16:11:43.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
16:11:43.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
16:11:43.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:11:43.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:11:43.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:11:43.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-2/logs
16:11:43.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-2/data/info.pb
16:11:43.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:11:43.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
16:11:43.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
16:11:43.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
16:11:43.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
16:11:43.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:11:43.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:11:43.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:11:43.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
16:11:43.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
16:11:43.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.234.131:42843
16:11:43.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.16.234.131
16:11:43.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.16.234.190:33901,127.16.234.189:32987,127.16.234.188:38065
16:11:43.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=35277
16:11:43.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.234.131
16:11:43.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.234.148:43519
16:11:43.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:11:43.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:11:43.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
16:11:43.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:43.790437 19962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
16:11:43.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:43.790931 19962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
16:11:43.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:43.791363 19962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
16:11:43.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:43.822674 19962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
16:11:43.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:43.823480 19962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.234.131
16:11:43.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:43.860468 19962 tablet_server_runner.cc:78] Tablet server non-default flags:
16:11:43.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:11:43.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.234.148:43519
16:11:43.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:11:43.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
16:11:43.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-2/data
16:11:43.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-2/wal
16:11:43.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:11:43.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:11:43.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.234.131:42843
16:11:43.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
16:11:43.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:11:43.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:11:43.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-2/data/info.pb
16:11:43.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.234.131
16:11:43.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=35277
16:11:43.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.16.234.190:33901,127.16.234.189:32987,127.16.234.188:38065
16:11:43.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
16:11:43.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:11:43.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
16:11:43.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
16:11:43.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:11:43.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.16.234.131
16:11:43.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-2/logs
16:11:43.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:11:43.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
16:11:43.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
16:11:43.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
16:11:43.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
16:11:43.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
16:11:43.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
16:11:43.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 15:12:52 UTC on ccycloud.quasar-tzrfpd.root.comops.site
16:11:43.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4580143
16:11:43.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
16:11:43.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:43.862046 19962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
16:11:43.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:43.864122 19962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
16:11:43.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:43.883608 19974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:11:45.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:43.884872 19973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:11:45.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:43.885730 19962 server_base.cc:1034] running on GCE node
16:11:45.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:43.885648 19976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:11:45.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:45.004669 19962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
16:11:45.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:45.007135 19962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
16:11:45.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:45.008528 19962 hybrid_clock.cc:648] HybridClock initialized: now 1786464705008487 us; error 37 us; skew 500 ppm
16:11:45.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:45.009410 19962 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
16:11:45.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:45.016145 19962 webserver.cc:460] Webserver started at http://127.16.234.131:35277/ using document root <none> and password file <none>
16:11:45.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:45.017108 19962 fs_manager.cc:359] Metadata directory not provided
16:11:45.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:45.017369 19962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
16:11:45.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:45.025199 19962 fs_manager.cc:709] Time spent opening directory manager: real 0.005s	user 0.006s	sys 0.000s
16:11:45.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:45.029626 19990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
16:11:45.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:45.030648 19962 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
16:11:45.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:45.030987 19962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-2/data,/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-2/wal
16:11:45.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "3d392ba816c24499a0f7fecd95f88c15"
16:11:45.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 16:11:18 on dist-test-slave-zj8q"
16:11:45.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:45.033345 19962 fs_report.cc:383] FS layout report
16:11:45.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
16:11:45.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-2/wal
16:11:45.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-2/wal
16:11:45.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-2/data/data
16:11:45.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
16:11:45.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
16:11:45.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
16:11:45.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
16:11:45.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
16:11:45.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
16:11:45.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
16:11:45.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
16:11:45.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
16:11:45.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
16:11:45.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
16:11:45.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:45.070375 19962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
16:11:45.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:45.071936 19962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
16:11:45.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:45.072417 19962 kserver.cc:163] Server-wide thread pool size limit: 3276
16:11:45.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:45.074970 19962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
16:11:45.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:45.080775 20004 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
16:11:45.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:45.088409 19962 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
16:11:45.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:45.088711 19962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s	user 0.001s	sys 0.001s
16:11:45.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:45.089002 19962 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
16:11:45.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:45.093637 19962 ts_tablet_manager.cc:610] Registered 1 tablets
16:11:45.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:45.093815 19962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s	user 0.002s	sys 0.000s
16:11:45.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:45.094218 20004 tablet_bootstrap.cc:492] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15: Bootstrap starting.
16:11:45.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:45.155762 20004 log.cc:824] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15: Log is configured to *not* fsync() on all Append() calls
16:11:45.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:45.228114 20004 tablet_bootstrap.cc:492] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15: Bootstrap replayed 1/1 log segments. Stats: ops{read=20 overwritten=0 applied=20 ignored=0} inserts{seen=15 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
16:11:45.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:45.228996 20004 tablet_bootstrap.cc:492] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15: Bootstrap complete.
16:11:45.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:45.230721 20004 ts_tablet_manager.cc:1397] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15: Time spent bootstrapping tablet: real 0.137s	user 0.088s	sys 0.046s
16:11:45.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:45.253923 20004 raft_consensus.cc:348] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 6 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:11:45.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:45.256930 20004 raft_consensus.cc:729] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d392ba816c24499a0f7fecd95f88c15, State: Initialized, Role: FOLLOWER
16:11:45.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:45.258205 20004 consensus_queue.cc:260] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 20, Last appended: 6.20, Last appended by leader: 20, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:11:45.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:45.262800 20004 ts_tablet_manager.cc:1428] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15: Time spent starting tablet: real 0.032s	user 0.029s	sys 0.001s
16:11:45.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:45.273181 19962 rpc_server.cc:307] RPC server started. Bound to: 127.16.234.131:42843
16:11:45.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:45.273443 20175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.234.131:42843 every 8 connection(s)
16:11:45.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:45.275641 19962 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-2/data/info.pb
16:11:45.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:45.281571 17322 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu as pid 19962
16:11:45.282 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
16:11:45.289 [ERROR - Time-limited test] (ResourceLeakDetector.java:327) LEAK: HashedWheelTimer.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
Created at:
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:312)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:251)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:224)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:203)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:185)
	org.apache.kudu.client.AsyncKuduScanner.<init>(AsyncKuduScanner.java:296)
	org.apache.kudu.client.AsyncKuduScanner.<init>(AsyncKuduScanner.java:431)
	org.apache.kudu.client.AsyncKuduScanner$AsyncKuduScannerBuilder.build(AsyncKuduScanner.java:1417)
	org.apache.kudu.client.TestMultipleLeaderFailover$1.get(TestMultipleLeaderFailover.java:48)
	org.apache.kudu.test.junit.AssertHelpers.assertEventuallyTrue(AssertHelpers.java:40)
	org.apache.kudu.client.TestMultipleLeaderFailover.waitUntilRowCount(TestMultipleLeaderFailover.java:44)
	org.apache.kudu.client.TestMultipleLeaderFailover.testMultipleFailover(TestMultipleLeaderFailover.java:79)
	sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	java.lang.reflect.Method.invoke(Method.java:498)
	org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
	org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
	java.util.concurrent.FutureTask.run(FutureTask.java:266)
	java.lang.Thread.run(Thread.java:750)
16:11:45.290 [ERROR - Time-limited test] (ResourceLeakDetector.java:327) LEAK: HashedWheelTimer.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
Created at:
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:312)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:251)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:224)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:203)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:185)
	org.apache.kudu.client.AsyncKuduScanner.<init>(AsyncKuduScanner.java:296)
	org.apache.kudu.client.AsyncKuduScanner.<init>(AsyncKuduScanner.java:431)
	org.apache.kudu.client.AsyncKuduScanner$AsyncKuduScannerBuilder.build(AsyncKuduScanner.java:1417)
	org.apache.kudu.client.TestMultipleLeaderFailover$1.get(TestMultipleLeaderFailover.java:48)
	org.apache.kudu.test.junit.AssertHelpers.assertEventuallyTrue(AssertHelpers.java:40)
	org.apache.kudu.client.TestMultipleLeaderFailover.waitUntilRowCount(TestMultipleLeaderFailover.java:44)
	org.apache.kudu.client.TestMultipleLeaderFailover.testMultipleFailover(TestMultipleLeaderFailover.java:97)
	sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	java.lang.reflect.Method.invoke(Method.java:498)
	org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
	org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
	java.util.concurrent.FutureTask.run(FutureTask.java:266)
	java.lang.Thread.run(Thread.java:750)
16:11:45.310 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 18 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304, tablet=7d3c128d101444a59e009013a5c634dc, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
16:11:45.312 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.16.234.130:43937
16:11:45.313 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

16:11:45.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:45.313887 17322 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu with pid 19655
16:11:45.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:45.357189 20179 heartbeater.cc:347] Connected to a master server at 127.16.234.190:33901
16:11:45.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:45.357652 20179 heartbeater.cc:464] Registering TS with master...
16:11:45.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:45.358723 20179 heartbeater.cc:511] Master 127.16.234.190:33901 requested a full tablet report, sending...
16:11:45.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:45.361385 20177 heartbeater.cc:347] Connected to a master server at 127.16.234.188:38065
16:11:45.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:45.361722 20177 heartbeater.cc:464] Registering TS with master...
16:11:45.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:45.362843 20177 heartbeater.cc:511] Master 127.16.234.188:38065 requested a full tablet report, sending...
16:11:45.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:45.363341 17401 ts_manager.cc:194] Re-registered known tserver with Master: 3d392ba816c24499a0f7fecd95f88c15 (127.16.234.131:42843)
16:11:45.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:45.365840 17659 ts_manager.cc:194] Re-registered known tserver with Master: 3d392ba816c24499a0f7fecd95f88c15 (127.16.234.131:42843)
16:11:45.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:45.369135 17659 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.234.131:35525
16:11:45.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:45.370211 20178 heartbeater.cc:347] Connected to a master server at 127.16.234.189:32987
16:11:45.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:45.370519 20178 heartbeater.cc:464] Registering TS with master...
16:11:45.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:45.371297 20178 heartbeater.cc:511] Master 127.16.234.189:32987 requested a full tablet report, sending...
16:11:45.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:45.376107 17536 ts_manager.cc:194] Re-registered known tserver with Master: 3d392ba816c24499a0f7fecd95f88c15 (127.16.234.131:42843)
16:11:45.391 [DEBUG - kudu-nio-1] (Connection.java:239) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] handling channelInactive
16:11:45.391 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] cleaning up while in state READY due to: connection closed
16:11:45.391 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
16:11:45.392 [DEBUG - Time-limited test] (Connection.java:748) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] connecting to peer
16:11:45.393 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:11:45.393 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:11:45.394 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:11:45.400 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:11:45.400 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)[L],3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843),d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)]}]
16:11:45.401 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=2, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=9), Traces: [0ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [2ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [2ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [8ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@1881764225(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@67b9eec9 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7ec39c3f -> wakeup thread Time-limited test)) after lookup
16:11:45.401 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] connecting to peer
16:11:45.402 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:11:45.402 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:11:45.402 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:11:45.420 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:11:45.421 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)[L],3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843),d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)]}]
16:11:45.421 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=4, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=29), Traces: [0ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [2ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [2ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [8ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [9ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [10ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [10ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [28ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@1881764225(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@67b9eec9 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1696015274 after retry RPC@1865168157), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7ec39c3f -> wakeup thread Time-limited test -> (continuation of Deferred@1696015274 after retry RPC@1865168157))) after lookup
16:11:45.422 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] connecting to peer
16:11:45.424 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:11:45.424 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:11:45.424 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:11:45.440 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:11:45.440 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)[L],3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843),d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)]}]
16:11:45.440 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=6, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=48), Traces: [0ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [2ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [2ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [8ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [9ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [10ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [10ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [28ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [30ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [32ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [32ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [44ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [47ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@1881764225(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@67b9eec9 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1696015274 after retry RPC@1865168157) -> (continuation of Deferred@1403581122 after retry RPC@1570939255), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7ec39c3f -> wakeup thread Time-limited test -> (continuation of Deferred@1696015274 after retry RPC@1865168157) -> (continuation of Deferred@1403581122 after retry RPC@1570939255))) after lookup
16:11:45.441 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] connecting to peer
16:11:45.442 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:11:45.442 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:11:45.442 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:11:45.540 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:11:45.541 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)[L],3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843),d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)]}]
16:11:45.541 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=8, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=149), Traces: [0ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [2ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [2ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [8ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [9ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [10ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [10ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [28ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [30ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [32ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [32ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [44ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [47ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [49ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [50ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [50ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [144ms] refreshing cache from master, [145ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [148ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@1881764225(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@67b9eec9 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1696015274 after retry RPC@1865168157) -> (continuation of Deferred@1403581122 after retry RPC@1570939255) -> (continuation of Deferred@448020801 after retry RPC@1737644058), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7ec39c3f -> wakeup thread Time-limited test -> (continuation of Deferred@1696015274 after retry RPC@1865168157) -> (continuation of Deferred@1403581122 after retry RPC@1570939255) -> (continuation of Deferred@448020801 after retry RPC@1737644058))) after lookup
16:11:45.543 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] connecting to peer
16:11:45.544 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:11:45.544 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:11:45.544 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:11:45.861 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:11:45.862 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)[L],3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843),d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)]}]
16:11:45.862 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=10, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=470), Traces: [0ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [2ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [2ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [8ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [9ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [10ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [10ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [28ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [30ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [32ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [32ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [44ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [47ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [49ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [50ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [50ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [144ms] refreshing cache from master, [145ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [148ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [150ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [152ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [153ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [465ms] refreshing cache from master, [465ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [469ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@1881764225(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@67b9eec9 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1696015274 after retry RPC@1865168157) -> (continuation of Deferred@1403581122 after retry RPC@1570939255) -> (continuation of Deferred@448020801 after retry RPC@1737644058) -> (continuation of Deferred@443792994 after retry RPC@1876195262), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7ec39c3f -> wakeup thread Time-limited test -> (continuation of Deferred@1696015274 after retry RPC@1865168157) -> (continuation of Deferred@1403581122 after retry RPC@1570939255) -> (continuation of Deferred@448020801 after retry RPC@1737644058) -> (continuation of Deferred@443792994 after retry RPC@1876195262))) after lookup
16:11:45.863 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] connecting to peer
16:11:45.864 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:11:45.864 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:11:45.864 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:11:46.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:46.374191 20177 heartbeater.cc:503] Master 127.16.234.188:38065 was elected leader, sending a full tablet report...
16:11:46.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:46.614853 20190 raft_consensus.cc:482] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1bce670edc6c40b2bb1584ec3b364e0f)
16:11:46.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:46.615386 20190 raft_consensus.cc:504] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:11:46.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:46.617414 20190 leader_election.cc:288] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3d392ba816c24499a0f7fecd95f88c15 (127.16.234.131:42843), 1bce670edc6c40b2bb1584ec3b364e0f (127.16.234.130:43937)
16:11:46.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:46.623164 19382 leader_election.cc:334] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 1bce670edc6c40b2bb1584ec3b364e0f (127.16.234.130:43937): Network error: Client connection negotiation failed: client connection to 127.16.234.130:43937: connect: Connection refused (error 111)
16:11:46.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:46.630045 20105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d3c128d101444a59e009013a5c634dc" candidate_uuid: "d9315253ea9e464c9416773b52ee00a6" candidate_term: 8 candidate_status { last_received { term: 7 index: 24 } } ignore_live_leader: false dest_uuid: "3d392ba816c24499a0f7fecd95f88c15" is_pre_election: true
16:11:46.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:46.630769 20105 raft_consensus.cc:2454] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9315253ea9e464c9416773b52ee00a6 in term 6.
16:11:46.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:46.631916 19378 leader_election.cc:302] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d392ba816c24499a0f7fecd95f88c15, d9315253ea9e464c9416773b52ee00a6; no voters: 1bce670edc6c40b2bb1584ec3b364e0f
16:11:46.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:46.632534 20190 raft_consensus.cc:2781] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 7 FOLLOWER]: Leader pre-election won for term 8
16:11:46.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:46.632795 20190 raft_consensus.cc:482] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 1bce670edc6c40b2bb1584ec3b364e0f)
16:11:46.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:46.633001 20190 raft_consensus.cc:3037] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 7 FOLLOWER]: Advancing to term 8
16:11:46.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:46.637727 20190 raft_consensus.cc:504] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:11:46.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:46.639050 20190 leader_election.cc:288] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [CANDIDATE]: Term 8 election: Requested vote from peers 3d392ba816c24499a0f7fecd95f88c15 (127.16.234.131:42843), 1bce670edc6c40b2bb1584ec3b364e0f (127.16.234.130:43937)
16:11:46.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:46.639780 20105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d3c128d101444a59e009013a5c634dc" candidate_uuid: "d9315253ea9e464c9416773b52ee00a6" candidate_term: 8 candidate_status { last_received { term: 7 index: 24 } } ignore_live_leader: false dest_uuid: "3d392ba816c24499a0f7fecd95f88c15"
16:11:46.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:46.640239 20105 raft_consensus.cc:3037] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 6 FOLLOWER]: Advancing to term 8
16:11:46.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:46.642072 19382 leader_election.cc:334] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 1bce670edc6c40b2bb1584ec3b364e0f (127.16.234.130:43937): Network error: Client connection negotiation failed: client connection to 127.16.234.130:43937: connect: Connection refused (error 111)
16:11:46.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:46.647136 20105 raft_consensus.cc:2454] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9315253ea9e464c9416773b52ee00a6 in term 8.
16:11:46.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:46.647897 19378 leader_election.cc:302] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d392ba816c24499a0f7fecd95f88c15, d9315253ea9e464c9416773b52ee00a6; no voters: 1bce670edc6c40b2bb1584ec3b364e0f
16:11:46.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:46.648473 20190 raft_consensus.cc:2781] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 8 FOLLOWER]: Leader election won for term 8
16:11:46.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:46.649462 20190 raft_consensus.cc:686] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 8 LEADER]: Becoming Leader. State: Replica: d9315253ea9e464c9416773b52ee00a6, State: Running, Role: LEADER
16:11:46.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:46.650627 20190 consensus_queue.cc:237] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 7.24, Last appended by leader: 24, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:11:46.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:46.660207 17662 catalog_manager.cc:5495] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 reported cstate change: term changed from 7 to 8, leader changed from 1bce670edc6c40b2bb1584ec3b364e0f (127.16.234.130) to d9315253ea9e464c9416773b52ee00a6 (127.16.234.129). New cstate: current_term: 8 leader_uuid: "d9315253ea9e464c9416773b52ee00a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } health_report { overall_health: UNKNOWN } } }
16:11:46.662 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:11:46.663 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)[L],3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843),d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)]}]
16:11:46.663 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=12, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=1271), Traces: [0ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [2ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [2ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [8ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [9ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [10ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [10ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [28ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [30ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [32ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [32ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [44ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [47ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [49ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [50ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [50ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [144ms] refreshing cache from master, [145ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [148ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [150ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [152ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [153ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [465ms] refreshing cache from master, [465ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [469ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [471ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [472ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [472ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [1264ms] refreshing cache from master, [1264ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [1270ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@1881764225(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@67b9eec9 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1696015274 after retry RPC@1865168157) -> (continuation of Deferred@1403581122 after retry RPC@1570939255) -> (continuation of Deferred@448020801 after retry RPC@1737644058) -> (continuation of Deferred@443792994 after retry RPC@1876195262) -> (continuation of Deferred@1247161496 after retry RPC@292508591), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7ec39c3f -> wakeup thread Time-limited test -> (continuation of Deferred@1696015274 after retry RPC@1865168157) -> (continuation of Deferred@1403581122 after retry RPC@1570939255) -> (continuation of Deferred@448020801 after retry RPC@1737644058) -> (continuation of Deferred@443792994 after retry RPC@1876195262) -> (continuation of Deferred@1247161496 after retry RPC@292508591))) after lookup
16:11:46.665 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] connecting to peer
16:11:46.667 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:11:46.667 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:11:46.667 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:11:47.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:47.078552 19382 consensus_peers.cc:487] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 -> Peer 1bce670edc6c40b2bb1584ec3b364e0f (127.16.234.130:43937): Couldn't send request to peer 1bce670edc6c40b2bb1584ec3b364e0f. Status: Network error: Client connection negotiation failed: client connection to 127.16.234.130:43937: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
16:11:47.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:47.215024 20105 raft_consensus.cc:1261] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 8 FOLLOWER]: Refusing update from remote peer d9315253ea9e464c9416773b52ee00a6: Log matching property violated. Preceding OpId in replica: term: 6 index: 20. Preceding OpId from leader: term: 8 index: 25. (index mismatch)
16:11:47.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:47.216878 20190 consensus_queue.cc:1035] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 20, Time since last communication: 0.000s
16:11:48.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:48.065240 19382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.234.130:43937: connect: Connection refused (error 111) [suppressed 4 similar messages]
16:11:48.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:48.239672 17703 debug-util.cc:398] Leaking SignalData structure 0x7b08000b7fe0 after lost signal to thread 17609
16:11:48.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:48.240180 17703 debug-util.cc:398] Leaking SignalData structure 0x7b08000b4060 after lost signal to thread 17710
16:11:48.340 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:11:48.340 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937),3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843),d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)[L]]}]
16:11:48.341 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=14, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=2949), Traces: [0ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [2ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [2ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [8ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [9ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [10ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [10ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [28ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [30ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [32ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [32ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [44ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [47ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [49ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [50ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [50ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [144ms] refreshing cache from master, [145ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [148ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [150ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [152ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [153ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [465ms] refreshing cache from master, [465ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [469ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [471ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [472ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [472ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [1264ms] refreshing cache from master, [1264ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [1270ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [1272ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [1275ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [1275ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [2944ms] refreshing cache from master, [2944ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [2947ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@1881764225(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@67b9eec9 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1696015274 after retry RPC@1865168157) -> (continuation of Deferred@1403581122 after retry RPC@1570939255) -> (continuation of Deferred@448020801 after retry RPC@1737644058) -> (continuation of Deferred@443792994 after retry RPC@1876195262) -> (continuation of Deferred@1247161496 after retry RPC@292508591) -> (continuation of Deferred@157483519 after retry RPC@1652214103), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7ec39c3f -> wakeup thread Time-limited test -> (continuation of Deferred@1696015274 after retry RPC@1865168157) -> (continuation of Deferred@1403581122 after retry RPC@1570939255) -> (continuation of Deferred@448020801 after retry RPC@1737644058) -> (continuation of Deferred@443792994 after retry RPC@1876195262) -> (continuation of Deferred@1247161496 after retry RPC@292508591) -> (continuation of Deferred@157483519 after retry RPC@1652214103))) after lookup
16:11:48.342 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)] connecting to peer
16:11:48.343 [DEBUG - kudu-nio-1] (Connection.java:757) [peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)] Successfully connected to peer
16:11:48.372 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0xb87d9701, L:/127.0.0.1:40702 - R:/127.16.234.129:42901] using TOKEN/null
16:11:48.416 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.16.234.130:43937
16:11:48.417 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

16:11:48.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:48.418742 17322 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu
16:11:48.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu
16:11:48.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/wal
16:11:48.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/data
16:11:48.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
16:11:48.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
16:11:48.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
16:11:48.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:11:48.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:11:48.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:11:48.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/logs
16:11:48.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/data/info.pb
16:11:48.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:11:48.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
16:11:48.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
16:11:48.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
16:11:48.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
16:11:48.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:11:48.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:11:48.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:11:48.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
16:11:48.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
16:11:48.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.234.130:43937
16:11:48.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.16.234.130
16:11:48.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.16.234.190:33901,127.16.234.189:32987,127.16.234.188:38065
16:11:48.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=37863
16:11:48.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.234.130
16:11:48.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.234.148:43519
16:11:48.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:11:48.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:11:48.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
16:11:48.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:48.722713 20268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
16:11:48.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:48.723189 20268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
16:11:48.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:48.723618 20268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
16:11:48.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:48.754634 20268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
16:11:48.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:48.755441 20268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.234.130
16:11:48.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:48.789548 20268 tablet_server_runner.cc:78] Tablet server non-default flags:
16:11:48.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:11:48.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.234.148:43519
16:11:48.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:11:48.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
16:11:48.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/data
16:11:48.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/wal
16:11:48.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:11:48.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:11:48.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.234.130:43937
16:11:48.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
16:11:48.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:11:48.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:11:48.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/data/info.pb
16:11:48.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.234.130
16:11:48.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=37863
16:11:48.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.16.234.190:33901,127.16.234.189:32987,127.16.234.188:38065
16:11:48.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
16:11:48.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:11:48.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
16:11:48.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
16:11:48.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:11:48.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.16.234.130
16:11:48.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/logs
16:11:48.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:11:48.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
16:11:48.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
16:11:48.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
16:11:48.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
16:11:48.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
16:11:48.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
16:11:48.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 15:12:52 UTC on ccycloud.quasar-tzrfpd.root.comops.site
16:11:48.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4580143
16:11:48.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
16:11:48.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:48.790776 20268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
16:11:48.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:48.792344 20268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
16:11:48.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:48.811983 20277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:11:49.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:49.787429 19382 consensus_peers.cc:487] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 -> Peer 1bce670edc6c40b2bb1584ec3b364e0f (127.16.234.130:43937): Couldn't send request to peer 1bce670edc6c40b2bb1584ec3b364e0f. Status: Network error: Client connection negotiation failed: client connection to 127.16.234.130:43937: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
16:11:49.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:48.812423 20278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:11:49.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:48.814270 20268 server_base.cc:1034] running on GCE node
16:11:49.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:48.812636 20281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:11:49.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:49.919713 20268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
16:11:49.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:49.922426 20268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
16:11:49.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:49.923849 20268 hybrid_clock.cc:648] HybridClock initialized: now 1786464709923822 us; error 47 us; skew 500 ppm
16:11:49.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:49.924682 20268 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
16:11:49.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:49.936190 20268 webserver.cc:460] Webserver started at http://127.16.234.130:37863/ using document root <none> and password file <none>
16:11:49.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:49.937199 20268 fs_manager.cc:359] Metadata directory not provided
16:11:49.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:49.937409 20268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
16:11:49.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:49.945120 20268 fs_manager.cc:709] Time spent opening directory manager: real 0.005s	user 0.006s	sys 0.000s
16:11:49.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:49.949424 20299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
16:11:49.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:49.950538 20268 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
16:11:49.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:49.950821 20268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/data,/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/wal
16:11:49.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "1bce670edc6c40b2bb1584ec3b364e0f"
16:11:49.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 16:11:16 on dist-test-slave-zj8q"
16:11:49.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:49.954267 20268 fs_report.cc:383] FS layout report
16:11:49.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
16:11:49.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/wal
16:11:49.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/wal
16:11:49.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/data/data
16:11:49.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
16:11:49.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
16:11:49.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
16:11:49.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
16:11:49.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
16:11:49.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
16:11:49.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
16:11:49.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
16:11:49.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
16:11:49.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
16:11:49.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
16:11:49.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:49.986219 20268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
16:11:49.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:49.987941 20268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
16:11:49.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:49.988499 20268 kserver.cc:163] Server-wide thread pool size limit: 3276
16:11:49.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:49.991078 20268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
16:11:49.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:49.996776 20315 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
16:11:50.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:50.004447 20268 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
16:11:50.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:50.004663 20268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s	user 0.001s	sys 0.000s
16:11:50.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:50.004889 20268 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
16:11:50.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:50.009745 20268 ts_tablet_manager.cc:610] Registered 1 tablets
16:11:50.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:50.009929 20268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s	user 0.002s	sys 0.000s
16:11:50.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:50.010317 20315 tablet_bootstrap.cc:492] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f: Bootstrap starting.
16:11:50.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:50.072773 20315 log.cc:824] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f: Log is configured to *not* fsync() on all Append() calls
16:11:50.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:50.191514 20315 tablet_bootstrap.cc:492] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f: Bootstrap replayed 1/1 log segments. Stats: ops{read=24 overwritten=0 applied=24 ignored=0} inserts{seen=18 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
16:11:50.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:50.192354 20315 tablet_bootstrap.cc:492] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f: Bootstrap complete.
16:11:50.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:50.193723 20315 ts_tablet_manager.cc:1397] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f: Time spent bootstrapping tablet: real 0.184s	user 0.124s	sys 0.050s
16:11:50.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:50.221130 20315 raft_consensus.cc:348] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [term 7 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:11:50.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:50.224192 20315 raft_consensus.cc:729] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bce670edc6c40b2bb1584ec3b364e0f, State: Initialized, Role: FOLLOWER
16:11:50.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:50.225425 20315 consensus_queue.cc:260] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 24, Last appended: 7.24, Last appended by leader: 24, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:11:50.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:50.235997 20315 ts_tablet_manager.cc:1428] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f: Time spent starting tablet: real 0.042s	user 0.036s	sys 0.009s
16:11:50.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:50.347189 20268 rpc_server.cc:307] RPC server started. Bound to: 127.16.234.130:43937
16:11:50.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:50.348455 20496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.234.130:43937 every 8 connection(s)
16:11:50.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:50.351338 20268 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/data/info.pb
16:11:50.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:50.361585 17322 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu as pid 20268
16:11:50.363 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
16:11:50.419 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 21 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304, tablet=7d3c128d101444a59e009013a5c634dc, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
16:11:50.420 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.16.234.129:42901
16:11:50.420 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

16:11:50.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:50.421367 17322 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu with pid 19349
16:11:50.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:50.475608 20498 heartbeater.cc:347] Connected to a master server at 127.16.234.188:38065
16:11:50.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:50.476049 20498 heartbeater.cc:464] Registering TS with master...
16:11:50.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:50.477279 20498 heartbeater.cc:511] Master 127.16.234.188:38065 requested a full tablet report, sending...
16:11:50.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:50.481421 17659 ts_manager.cc:194] Re-registered known tserver with Master: 1bce670edc6c40b2bb1584ec3b364e0f (127.16.234.130:43937)
16:11:50.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:50.484395 17659 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.234.130:47559
16:11:50.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:50.488258 20507 heartbeater.cc:347] Connected to a master server at 127.16.234.190:33901
16:11:50.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:50.488593 20507 heartbeater.cc:464] Registering TS with master...
16:11:50.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:50.489251 20507 heartbeater.cc:511] Master 127.16.234.190:33901 requested a full tablet report, sending...
16:11:50.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:50.491400 20499 heartbeater.cc:347] Connected to a master server at 127.16.234.189:32987
16:11:50.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:50.491719 20499 heartbeater.cc:464] Registering TS with master...
16:11:50.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:50.492331 20499 heartbeater.cc:511] Master 127.16.234.189:32987 requested a full tablet report, sending...
16:11:50.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:50.494855 17402 ts_manager.cc:194] Re-registered known tserver with Master: 1bce670edc6c40b2bb1584ec3b364e0f (127.16.234.130:43937)
16:11:50.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:50.496999 17537 ts_manager.cc:194] Re-registered known tserver with Master: 1bce670edc6c40b2bb1584ec3b364e0f (127.16.234.130:43937)
16:11:50.506 [DEBUG - kudu-nio-1] (Connection.java:239) [peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)] handling channelInactive
16:11:50.506 [DEBUG - kudu-nio-1] (Connection.java:709) [peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)] cleaning up while in state READY due to: connection closed
16:11:50.508 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
16:11:50.509 [DEBUG - Time-limited test] (Connection.java:748) [peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)] connecting to peer
16:11:50.514 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901
16:11:50.514 [DEBUG - kudu-nio-0] (Connection.java:709) [peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)] cleaning up while in state CONNECTING due to: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901
16:11:50.514 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901
16:11:50.529 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:11:50.529 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937),3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843),d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)[L]]}]
16:11:50.530 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=2, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=21), Traces: [0ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [6ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [6ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [21ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@49160032(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@46c37fc7 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4de99bd -> wakeup thread Time-limited test)) after lookup
16:11:50.530 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)] connecting to peer
16:11:50.531 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901
16:11:50.531 [DEBUG - kudu-nio-1] (Connection.java:709) [peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)] cleaning up while in state CONNECTING due to: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901
16:11:50.531 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901
16:11:50.561 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:11:50.561 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937),3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843),d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)[L]]}]
16:11:50.561 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=4, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=53), Traces: [0ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [6ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [6ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [21ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [22ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [23ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [23ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [52ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@49160032(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@46c37fc7 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@216661986 after retry RPC@72447999), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4de99bd -> wakeup thread Time-limited test -> (continuation of Deferred@216661986 after retry RPC@72447999))) after lookup
16:11:50.562 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)] connecting to peer
16:11:50.564 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901
16:11:50.564 [DEBUG - kudu-nio-0] (Connection.java:709) [peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)] cleaning up while in state CONNECTING due to: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901
16:11:50.564 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901
16:11:50.601 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:11:50.601 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937),3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843),d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)[L]]}]
16:11:50.601 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=6, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=93), Traces: [0ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [6ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [6ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [21ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [22ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [23ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [23ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [52ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [54ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [56ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [56ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [88ms] refreshing cache from master, [88ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [92ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@49160032(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@46c37fc7 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@216661986 after retry RPC@72447999) -> (continuation of Deferred@748986256 after retry RPC@591788916), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4de99bd -> wakeup thread Time-limited test -> (continuation of Deferred@216661986 after retry RPC@72447999) -> (continuation of Deferred@748986256 after retry RPC@591788916))) after lookup
16:11:50.602 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)] connecting to peer
16:11:50.603 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901
16:11:50.603 [DEBUG - kudu-nio-1] (Connection.java:709) [peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)] cleaning up while in state CONNECTING due to: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901
16:11:50.603 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901
16:11:50.700 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:11:50.700 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937),3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843),d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)[L]]}]
16:11:50.701 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=8, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=192), Traces: [0ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [6ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [6ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [21ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [22ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [23ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [23ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [52ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [54ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [56ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [56ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [88ms] refreshing cache from master, [88ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [92ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [93ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [95ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [95ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [188ms] refreshing cache from master, [188ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [192ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@49160032(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@46c37fc7 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@216661986 after retry RPC@72447999) -> (continuation of Deferred@748986256 after retry RPC@591788916) -> (continuation of Deferred@364480053 after retry RPC@10488205), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4de99bd -> wakeup thread Time-limited test -> (continuation of Deferred@216661986 after retry RPC@72447999) -> (continuation of Deferred@748986256 after retry RPC@591788916) -> (continuation of Deferred@364480053 after retry RPC@10488205))) after lookup
16:11:50.702 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)] connecting to peer
16:11:50.704 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901
16:11:50.704 [DEBUG - kudu-nio-0] (Connection.java:709) [peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)] cleaning up while in state CONNECTING due to: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901
16:11:50.704 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901
16:11:50.980 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:11:50.980 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937),3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843),d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)[L]]}]
16:11:50.981 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=10, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=472), Traces: [0ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [6ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [6ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [21ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [22ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [23ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [23ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [52ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [54ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [56ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [56ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [88ms] refreshing cache from master, [88ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [92ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [93ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [95ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [95ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [188ms] refreshing cache from master, [188ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [192ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [194ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [196ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [196ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [468ms] refreshing cache from master, [469ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [472ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@49160032(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@46c37fc7 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@216661986 after retry RPC@72447999) -> (continuation of Deferred@748986256 after retry RPC@591788916) -> (continuation of Deferred@364480053 after retry RPC@10488205) -> (continuation of Deferred@1137336355 after retry RPC@1126622646), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4de99bd -> wakeup thread Time-limited test -> (continuation of Deferred@216661986 after retry RPC@72447999) -> (continuation of Deferred@748986256 after retry RPC@591788916) -> (continuation of Deferred@364480053 after retry RPC@10488205) -> (continuation of Deferred@1137336355 after retry RPC@1126622646))) after lookup
16:11:50.981 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)] connecting to peer
16:11:50.982 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901
16:11:50.982 [DEBUG - kudu-nio-1] (Connection.java:709) [peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)] cleaning up while in state CONNECTING due to: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901
16:11:50.983 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901
16:11:51.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:51.490480 20498 heartbeater.cc:503] Master 127.16.234.188:38065 was elected leader, sending a full tablet report...
16:11:51.540 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:11:51.540 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937),3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843),d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)[L]]}]
16:11:51.540 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=12, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=1032), Traces: [0ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [6ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [6ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [21ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [22ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [23ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [23ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [52ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [54ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [56ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [56ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [88ms] refreshing cache from master, [88ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [92ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [93ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [95ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [95ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [188ms] refreshing cache from master, [188ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [192ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [194ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [196ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [196ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [468ms] refreshing cache from master, [469ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [472ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [473ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [475ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [475ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [1028ms] refreshing cache from master, [1028ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [1031ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@49160032(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@46c37fc7 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@216661986 after retry RPC@72447999) -> (continuation of Deferred@748986256 after retry RPC@591788916) -> (continuation of Deferred@364480053 after retry RPC@10488205) -> (continuation of Deferred@1137336355 after retry RPC@1126622646) -> (continuation of Deferred@732641583 after retry RPC@2008890850), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4de99bd -> wakeup thread Time-limited test -> (continuation of Deferred@216661986 after retry RPC@72447999) -> (continuation of Deferred@748986256 after retry RPC@591788916) -> (continuation of Deferred@364480053 after retry RPC@10488205) -> (continuation of Deferred@1137336355 after retry RPC@1126622646) -> (continuation of Deferred@732641583 after retry RPC@2008890850))) after lookup
16:11:51.541 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)] connecting to peer
16:11:51.543 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901
16:11:51.543 [DEBUG - kudu-nio-0] (Connection.java:709) [peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)] cleaning up while in state CONNECTING due to: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901
16:11:51.543 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901
16:11:51.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:51.877688 20515 raft_consensus.cc:482] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d9315253ea9e464c9416773b52ee00a6)
16:11:51.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:51.878098 20515 raft_consensus.cc:504] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:11:51.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:51.880573 20515 leader_election.cc:288] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d9315253ea9e464c9416773b52ee00a6 (127.16.234.129:42901), 1bce670edc6c40b2bb1584ec3b364e0f (127.16.234.130:43937)
16:11:51.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:51.883405 19995 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.234.129:42901: connect: Connection refused (error 111)
16:11:51.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:51.887703 19995 leader_election.cc:334] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer d9315253ea9e464c9416773b52ee00a6 (127.16.234.129:42901): Network error: Client connection negotiation failed: client connection to 127.16.234.129:42901: connect: Connection refused (error 111)
16:11:51.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:51.892669 20419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d3c128d101444a59e009013a5c634dc" candidate_uuid: "3d392ba816c24499a0f7fecd95f88c15" candidate_term: 9 candidate_status { last_received { term: 8 index: 28 } } ignore_live_leader: false dest_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" is_pre_election: true
16:11:51.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:51.893355 20419 raft_consensus.cc:2454] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d392ba816c24499a0f7fecd95f88c15 in term 7.
16:11:51.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:51.894593 19996 leader_election.cc:302] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1bce670edc6c40b2bb1584ec3b364e0f, 3d392ba816c24499a0f7fecd95f88c15; no voters: d9315253ea9e464c9416773b52ee00a6
16:11:51.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:51.895246 20515 raft_consensus.cc:2781] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 8 FOLLOWER]: Leader pre-election won for term 9
16:11:51.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:51.895486 20515 raft_consensus.cc:482] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 8 FOLLOWER]: Starting leader election (detected failure of leader d9315253ea9e464c9416773b52ee00a6)
16:11:51.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:51.895694 20515 raft_consensus.cc:3037] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 8 FOLLOWER]: Advancing to term 9
16:11:51.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:51.899989 20515 raft_consensus.cc:504] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:11:51.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:51.901355 20515 leader_election.cc:288] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [CANDIDATE]: Term 9 election: Requested vote from peers d9315253ea9e464c9416773b52ee00a6 (127.16.234.129:42901), 1bce670edc6c40b2bb1584ec3b364e0f (127.16.234.130:43937)
16:11:51.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:51.902420 20419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d3c128d101444a59e009013a5c634dc" candidate_uuid: "3d392ba816c24499a0f7fecd95f88c15" candidate_term: 9 candidate_status { last_received { term: 8 index: 28 } } ignore_live_leader: false dest_uuid: "1bce670edc6c40b2bb1584ec3b364e0f"
16:11:51.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:51.902964 20419 raft_consensus.cc:3037] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [term 7 FOLLOWER]: Advancing to term 9
16:11:51.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:51.904340 19995 leader_election.cc:334] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer d9315253ea9e464c9416773b52ee00a6 (127.16.234.129:42901): Network error: Client connection negotiation failed: client connection to 127.16.234.129:42901: connect: Connection refused (error 111)
16:11:51.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:51.909739 20419 raft_consensus.cc:2454] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d392ba816c24499a0f7fecd95f88c15 in term 9.
16:11:51.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:51.910578 19996 leader_election.cc:302] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1bce670edc6c40b2bb1584ec3b364e0f, 3d392ba816c24499a0f7fecd95f88c15; no voters: d9315253ea9e464c9416773b52ee00a6
16:11:51.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:51.911128 20515 raft_consensus.cc:2781] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 9 FOLLOWER]: Leader election won for term 9
16:11:51.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:51.912590 20515 raft_consensus.cc:686] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 9 LEADER]: Becoming Leader. State: Replica: 3d392ba816c24499a0f7fecd95f88c15, State: Running, Role: LEADER
16:11:51.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:51.913442 20515 consensus_queue.cc:237] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 8.28, Last appended by leader: 28, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:11:51.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:51.922029 17659 catalog_manager.cc:5495] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 reported cstate change: term changed from 8 to 9, leader changed from d9315253ea9e464c9416773b52ee00a6 (127.16.234.129) to 3d392ba816c24499a0f7fecd95f88c15 (127.16.234.131). New cstate: current_term: 9 leader_uuid: "3d392ba816c24499a0f7fecd95f88c15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } health_report { overall_health: UNKNOWN } } }
16:11:51.944 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:11:51.944 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937),3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843),d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)[L]]}]
16:11:51.945 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=14, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=1436), Traces: [0ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [6ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [6ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [21ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [22ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [23ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [23ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [52ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [54ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [56ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [56ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [88ms] refreshing cache from master, [88ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [92ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [93ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [95ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [95ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [188ms] refreshing cache from master, [188ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [192ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [194ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [196ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [196ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [468ms] refreshing cache from master, [469ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [472ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [473ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [475ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [475ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [1028ms] refreshing cache from master, [1028ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [1031ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [1033ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [1035ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [1035ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [1428ms] refreshing cache from master, [1428ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [1436ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@49160032(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@46c37fc7 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@216661986 after retry RPC@72447999) -> (continuation of Deferred@748986256 after retry RPC@591788916) -> (continuation of Deferred@364480053 after retry RPC@10488205) -> (continuation of Deferred@1137336355 after retry RPC@1126622646) -> (continuation of Deferred@732641583 after retry RPC@2008890850) -> (continuation of Deferred@189344634 after retry RPC@361055708), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4de99bd -> wakeup thread Time-limited test -> (continuation of Deferred@216661986 after retry RPC@72447999) -> (continuation of Deferred@748986256 after retry RPC@591788916) -> (continuation of Deferred@364480053 after retry RPC@10488205) -> (continuation of Deferred@1137336355 after retry RPC@1126622646) -> (continuation of Deferred@732641583 after retry RPC@2008890850) -> (continuation of Deferred@189344634 after retry RPC@361055708))) after lookup
16:11:51.946 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)] connecting to peer
16:11:51.947 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901
16:11:51.947 [DEBUG - kudu-nio-1] (Connection.java:709) [peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)] cleaning up while in state CONNECTING due to: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901
16:11:51.947 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901
16:11:52.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:52.454989 20419 raft_consensus.cc:1261] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [term 9 FOLLOWER]: Refusing update from remote peer 3d392ba816c24499a0f7fecd95f88c15: Log matching property violated. Preceding OpId in replica: term: 7 index: 24. Preceding OpId from leader: term: 9 index: 29. (index mismatch)
16:11:52.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:52.456420 20554 consensus_queue.cc:1035] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 24, Time since last communication: 0.000s
16:11:52.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:52.505913 19995 consensus_peers.cc:487] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 -> Peer d9315253ea9e464c9416773b52ee00a6 (127.16.234.129:42901): Couldn't send request to peer d9315253ea9e464c9416773b52ee00a6. Status: Network error: Client connection negotiation failed: client connection to 127.16.234.129:42901: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
16:11:53.361 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:11:53.361 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937),3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)[L],d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)]}]
16:11:53.362 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=16, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=2853), Traces: [0ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [6ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [6ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [21ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [22ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [23ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [23ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [52ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [54ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [56ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [56ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [88ms] refreshing cache from master, [88ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [92ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [93ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [95ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [95ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [188ms] refreshing cache from master, [188ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [192ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [194ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [196ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [196ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [468ms] refreshing cache from master, [469ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [472ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [473ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [475ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [475ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [1028ms] refreshing cache from master, [1028ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [1031ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [1033ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [1035ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [1035ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [1428ms] refreshing cache from master, [1428ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [1436ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [1437ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [1439ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [1439ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [2849ms] refreshing cache from master, [2849ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [2852ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@49160032(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@46c37fc7 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@216661986 after retry RPC@72447999) -> (continuation of Deferred@748986256 after retry RPC@591788916) -> (continuation of Deferred@364480053 after retry RPC@10488205) -> (continuation of Deferred@1137336355 after retry RPC@1126622646) -> (continuation of Deferred@732641583 after retry RPC@2008890850) -> (continuation of Deferred@189344634 after retry RPC@361055708) -> (continuation of Deferred@803633306 after retry RPC@2004065834), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4de99bd -> wakeup thread Time-limited test -> (continuation of Deferred@216661986 after retry RPC@72447999) -> (continuation of Deferred@748986256 after retry RPC@591788916) -> (continuation of Deferred@364480053 after retry RPC@10488205) -> (continuation of Deferred@1137336355 after retry RPC@1126622646) -> (continuation of Deferred@732641583 after retry RPC@2008890850) -> (continuation of Deferred@189344634 after retry RPC@361055708) -> (continuation of Deferred@803633306 after retry RPC@2004065834))) after lookup
16:11:53.363 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)] connecting to peer
16:11:53.365 [DEBUG - kudu-nio-0] (Connection.java:757) [peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)] Successfully connected to peer
16:11:53.387 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0xe00a7f40, L:/127.0.0.1:54700 - R:/127.16.234.131:42843] using TOKEN/null
16:11:53.431 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.16.234.129:42901
16:11:53.432 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

16:11:53.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:53.433634 17322 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu
16:11:53.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu
16:11:53.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-0/wal
16:11:53.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-0/data
16:11:53.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
16:11:53.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
16:11:53.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
16:11:53.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:11:53.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:11:53.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:11:53.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-0/logs
16:11:53.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-0/data/info.pb
16:11:53.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:11:53.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
16:11:53.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
16:11:53.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
16:11:53.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
16:11:53.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:11:53.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:11:53.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:11:53.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
16:11:53.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
16:11:53.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.234.129:42901
16:11:53.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.16.234.129
16:11:53.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.16.234.190:33901,127.16.234.189:32987,127.16.234.188:38065
16:11:53.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=43191
16:11:53.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.234.129
16:11:53.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.234.148:43519
16:11:53.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:11:53.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:11:53.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
16:11:53.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:53.748607 20592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
16:11:53.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:53.749085 20592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
16:11:53.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:53.749522 20592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
16:11:53.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:53.784291 20592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
16:11:53.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:53.785112 20592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.234.129
16:11:53.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:53.824291 20592 tablet_server_runner.cc:78] Tablet server non-default flags:
16:11:53.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:11:53.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.234.148:43519
16:11:53.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:11:53.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
16:11:53.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-0/data
16:11:53.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-0/wal
16:11:53.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:11:53.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:11:53.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.234.129:42901
16:11:53.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
16:11:53.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:11:53.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:11:53.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-0/data/info.pb
16:11:53.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.234.129
16:11:53.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=43191
16:11:53.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.16.234.190:33901,127.16.234.189:32987,127.16.234.188:38065
16:11:53.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
16:11:53.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:11:53.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
16:11:53.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
16:11:53.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:11:53.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.16.234.129
16:11:53.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-0/logs
16:11:53.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:11:53.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
16:11:53.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
16:11:53.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
16:11:53.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
16:11:53.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
16:11:53.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
16:11:53.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 15:12:52 UTC on ccycloud.quasar-tzrfpd.root.comops.site
16:11:53.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4580143
16:11:53.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
16:11:53.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:53.825879 20592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
16:11:53.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:53.827836 20592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
16:11:53.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:53.846401 20606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:11:54.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:53.846448 20604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:11:54.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:53.847548 20603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:11:54.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:54.957484 20605 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
16:11:54.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:54.957580 20592 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
16:11:54.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:54.961499 20592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
16:11:54.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:54.963728 20592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
16:11:54.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:54.965065 20592 hybrid_clock.cc:648] HybridClock initialized: now 1786464714965034 us; error 49 us; skew 500 ppm
16:11:54.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:54.965852 20592 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
16:11:54.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:54.975719 20592 webserver.cc:460] Webserver started at http://127.16.234.129:43191/ using document root <none> and password file <none>
16:11:54.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:54.976590 20592 fs_manager.cc:359] Metadata directory not provided
16:11:54.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:54.976780 20592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
16:11:54.984 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:54.984686 20592 fs_manager.cc:709] Time spent opening directory manager: real 0.005s	user 0.003s	sys 0.002s
16:11:54.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:54.989049 20625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
16:11:54.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:54.990113 20592 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
16:11:54.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:54.990439 20592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-0/data,/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-0/wal
16:11:54.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "d9315253ea9e464c9416773b52ee00a6"
16:11:54.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 16:11:14 on dist-test-slave-zj8q"
16:11:54.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:54.992689 20592 fs_report.cc:383] FS layout report
16:11:54.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
16:11:54.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-0/wal
16:11:54.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-0/wal
16:11:54.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-0/data/data
16:11:54.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
16:11:54.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
16:11:54.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
16:11:54.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
16:11:54.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
16:11:54.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
16:11:54.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
16:11:54.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
16:11:54.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
16:11:54.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
16:11:54.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
16:11:55.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:55.039431 19995 consensus_peers.cc:487] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 -> Peer d9315253ea9e464c9416773b52ee00a6 (127.16.234.129:42901): Couldn't send request to peer d9315253ea9e464c9416773b52ee00a6. Status: Network error: Client connection negotiation failed: client connection to 127.16.234.129:42901: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
16:11:55.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:55.044602 20592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
16:11:55.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:55.046365 20592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
16:11:55.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:55.047046 20592 kserver.cc:163] Server-wide thread pool size limit: 3276
16:11:55.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:55.049785 20592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
16:11:55.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:55.055917 20633 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
16:11:55.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:55.063719 20592 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
16:11:55.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:55.063957 20592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s	user 0.002s	sys 0.000s
16:11:55.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:55.064298 20592 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
16:11:55.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:55.069069 20592 ts_tablet_manager.cc:610] Registered 1 tablets
16:11:55.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:55.069257 20592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s	user 0.000s	sys 0.003s
16:11:55.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:55.069558 20633 tablet_bootstrap.cc:492] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6: Bootstrap starting.
16:11:55.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:55.132583 20633 log.cc:824] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6: Log is configured to *not* fsync() on all Append() calls
16:11:55.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:55.227073 20633 tablet_bootstrap.cc:492] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=21 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
16:11:55.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:55.227878 20633 tablet_bootstrap.cc:492] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6: Bootstrap complete.
16:11:55.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:55.229169 20633 ts_tablet_manager.cc:1397] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6: Time spent bootstrapping tablet: real 0.160s	user 0.113s	sys 0.040s
16:11:55.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:55.243067 20592 rpc_server.cc:307] RPC server started. Bound to: 127.16.234.129:42901
16:11:55.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:55.243188 20822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.234.129:42901 every 8 connection(s)
16:11:55.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:55.245743 20592 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-0/data/info.pb
16:11:55.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:55.255966 17322 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu as pid 20592
16:11:55.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:55.248350 20633 raft_consensus.cc:348] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 8 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:11:55.256 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
16:11:55.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:55.256580 20633 raft_consensus.cc:729] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9315253ea9e464c9416773b52ee00a6, State: Initialized, Role: FOLLOWER
16:11:55.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:55.265967 20633 consensus_queue.cc:260] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 8.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:11:55.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:55.280136 20633 ts_tablet_manager.cc:1428] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6: Time spent starting tablet: real 0.051s	user 0.026s	sys 0.003s
16:11:55.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:55.294080 20824 heartbeater.cc:347] Connected to a master server at 127.16.234.188:38065
16:11:55.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:55.294605 20824 heartbeater.cc:464] Registering TS with master...
16:11:55.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:55.295770 20824 heartbeater.cc:511] Master 127.16.234.188:38065 requested a full tablet report, sending...
16:11:55.297 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 24 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304, tablet=7d3c128d101444a59e009013a5c634dc, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
16:11:55.298 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.16.234.131:42843
16:11:55.298 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

16:11:55.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:55.299017 17322 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu with pid 19962
16:11:55.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:55.300377 17659 ts_manager.cc:194] Re-registered known tserver with Master: d9315253ea9e464c9416773b52ee00a6 (127.16.234.129:42901)
16:11:55.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:55.305459 20826 heartbeater.cc:347] Connected to a master server at 127.16.234.190:33901
16:11:55.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:55.305774 20826 heartbeater.cc:464] Registering TS with master...
16:11:55.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:55.306452 20826 heartbeater.cc:511] Master 127.16.234.190:33901 requested a full tablet report, sending...
16:11:55.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:55.309278 17401 ts_manager.cc:194] Re-registered known tserver with Master: d9315253ea9e464c9416773b52ee00a6 (127.16.234.129:42901)
16:11:55.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:55.311065 17659 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.234.129:45669
16:11:55.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:55.317269 20825 heartbeater.cc:347] Connected to a master server at 127.16.234.189:32987
16:11:55.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:55.317612 20825 heartbeater.cc:464] Registering TS with master...
16:11:55.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:55.318439 20825 heartbeater.cc:511] Master 127.16.234.189:32987 requested a full tablet report, sending...
16:11:55.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:55.319171 20824 heartbeater.cc:503] Master 127.16.234.188:38065 was elected leader, sending a full tablet report...
16:11:55.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:55.321821 17537 ts_manager.cc:194] Re-registered known tserver with Master: d9315253ea9e464c9416773b52ee00a6 (127.16.234.129:42901)
16:11:55.348 [DEBUG - kudu-nio-0] (Connection.java:239) [peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)] handling channelInactive
16:11:55.348 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)] cleaning up while in state READY due to: connection closed
16:11:55.348 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
16:11:55.349 [DEBUG - Time-limited test] (Connection.java:748) [peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)] connecting to peer
16:11:55.350 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843
16:11:55.350 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843
16:11:55.351 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843
16:11:55.360 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:11:55.360 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937),3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)[L],d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)]}]
16:11:55.361 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=2, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=12), Traces: [0ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [3ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [3ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [12ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@1885593116(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@1cd73cb7 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@186d863c -> wakeup thread Time-limited test)) after lookup
16:11:55.361 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)] connecting to peer
16:11:55.363 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843
16:11:55.363 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843
16:11:55.363 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843
16:11:55.380 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:11:55.381 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937),3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)[L],d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)]}]
16:11:55.381 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=4, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=32), Traces: [0ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [3ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [3ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [12ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [13ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [15ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [15ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [32ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@1885593116(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@1cd73cb7 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1988583236 after retry RPC@36662222), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@186d863c -> wakeup thread Time-limited test -> (continuation of Deferred@1988583236 after retry RPC@36662222))) after lookup
16:11:55.382 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)] connecting to peer
16:11:55.383 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843
16:11:55.383 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843
16:11:55.383 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843
16:11:55.420 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:11:55.420 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937),3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)[L],d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)]}]
16:11:55.421 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=6, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=72), Traces: [0ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [3ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [3ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [12ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [13ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [15ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [15ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [32ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [34ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [35ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [35ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [68ms] refreshing cache from master, [69ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [72ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@1885593116(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@1cd73cb7 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1988583236 after retry RPC@36662222) -> (continuation of Deferred@803614473 after retry RPC@110988026), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@186d863c -> wakeup thread Time-limited test -> (continuation of Deferred@1988583236 after retry RPC@36662222) -> (continuation of Deferred@803614473 after retry RPC@110988026))) after lookup
16:11:55.421 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)] connecting to peer
16:11:55.422 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843
16:11:55.422 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843
16:11:55.423 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843
16:11:55.500 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:11:55.501 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937),3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)[L],d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)]}]
16:11:55.501 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=8, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=153), Traces: [0ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [3ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [3ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [12ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [13ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [15ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [15ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [32ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [34ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [35ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [35ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [68ms] refreshing cache from master, [69ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [72ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [73ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [75ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [75ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [148ms] refreshing cache from master, [148ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [152ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@1885593116(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@1cd73cb7 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1988583236 after retry RPC@36662222) -> (continuation of Deferred@803614473 after retry RPC@110988026) -> (continuation of Deferred@573870505 after retry RPC@1209275727), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@186d863c -> wakeup thread Time-limited test -> (continuation of Deferred@1988583236 after retry RPC@36662222) -> (continuation of Deferred@803614473 after retry RPC@110988026) -> (continuation of Deferred@573870505 after retry RPC@1209275727))) after lookup
16:11:55.502 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)] connecting to peer
16:11:55.503 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843
16:11:55.503 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843
16:11:55.503 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843
16:11:55.661 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:11:55.661 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937),3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)[L],d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)]}]
16:11:55.661 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=10, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=313), Traces: [0ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [3ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [3ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [12ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [13ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [15ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [15ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [32ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [34ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [35ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [35ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [68ms] refreshing cache from master, [69ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [72ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [73ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [75ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [75ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [148ms] refreshing cache from master, [148ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [152ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [154ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [155ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [155ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [309ms] refreshing cache from master, [309ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [312ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@1885593116(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@1cd73cb7 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1988583236 after retry RPC@36662222) -> (continuation of Deferred@803614473 after retry RPC@110988026) -> (continuation of Deferred@573870505 after retry RPC@1209275727) -> (continuation of Deferred@444862449 after retry RPC@1516415466), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@186d863c -> wakeup thread Time-limited test -> (continuation of Deferred@1988583236 after retry RPC@36662222) -> (continuation of Deferred@803614473 after retry RPC@110988026) -> (continuation of Deferred@573870505 after retry RPC@1209275727) -> (continuation of Deferred@444862449 after retry RPC@1516415466))) after lookup
16:11:55.662 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)] connecting to peer
16:11:55.664 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843
16:11:55.664 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843
16:11:55.664 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843
16:11:56.100 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:11:56.100 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937),3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)[L],d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)]}]
16:11:56.101 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=12, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=752), Traces: [0ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [3ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [3ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [12ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [13ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [15ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [15ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [32ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [34ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [35ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [35ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [68ms] refreshing cache from master, [69ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [72ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [73ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [75ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [75ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [148ms] refreshing cache from master, [148ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [152ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [154ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [155ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [155ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [309ms] refreshing cache from master, [309ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [312ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [314ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [316ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [316ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [748ms] refreshing cache from master, [748ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [751ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@1885593116(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@1cd73cb7 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1988583236 after retry RPC@36662222) -> (continuation of Deferred@803614473 after retry RPC@110988026) -> (continuation of Deferred@573870505 after retry RPC@1209275727) -> (continuation of Deferred@444862449 after retry RPC@1516415466) -> (continuation of Deferred@1041427155 after retry RPC@1911568642), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@186d863c -> wakeup thread Time-limited test -> (continuation of Deferred@1988583236 after retry RPC@36662222) -> (continuation of Deferred@803614473 after retry RPC@110988026) -> (continuation of Deferred@573870505 after retry RPC@1209275727) -> (continuation of Deferred@444862449 after retry RPC@1516415466) -> (continuation of Deferred@1041427155 after retry RPC@1911568642))) after lookup
16:11:56.101 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)] connecting to peer
16:11:56.102 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843
16:11:56.102 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843
16:11:56.102 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843
16:11:56.435 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:56.435374 20841 raft_consensus.cc:482] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
16:11:56.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:56.435931 20841 raft_consensus.cc:504] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:11:56.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:56.438479 20841 leader_election.cc:288] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3d392ba816c24499a0f7fecd95f88c15 (127.16.234.131:42843), 1bce670edc6c40b2bb1584ec3b364e0f (127.16.234.130:43937)
16:11:56.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:56.442539 20627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.234.131:42843: connect: Connection refused (error 111)
16:11:56.446 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:56.446447 20627 leader_election.cc:334] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 3d392ba816c24499a0f7fecd95f88c15 (127.16.234.131:42843): Network error: Client connection negotiation failed: client connection to 127.16.234.131:42843: connect: Connection refused (error 111)
16:11:56.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:56.451256 20419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d3c128d101444a59e009013a5c634dc" candidate_uuid: "d9315253ea9e464c9416773b52ee00a6" candidate_term: 9 candidate_status { last_received { term: 8 index: 28 } } ignore_live_leader: false dest_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" is_pre_election: true
16:11:56.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:56.452608 20630 leader_election.cc:302] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9315253ea9e464c9416773b52ee00a6; no voters: 1bce670edc6c40b2bb1584ec3b364e0f, 3d392ba816c24499a0f7fecd95f88c15
16:11:56.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:56.453287 20841 raft_consensus.cc:2726] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
16:11:56.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:56.655450 20850 raft_consensus.cc:482] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3d392ba816c24499a0f7fecd95f88c15)
16:11:56.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:56.655849 20850 raft_consensus.cc:504] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:11:56.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:56.658115 20850 leader_election.cc:288] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3d392ba816c24499a0f7fecd95f88c15 (127.16.234.131:42843), d9315253ea9e464c9416773b52ee00a6 (127.16.234.129:42901)
16:11:56.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:56.660374 20303 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.234.131:42843: connect: Connection refused (error 111)
16:11:56.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:56.664669 20303 leader_election.cc:334] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 3d392ba816c24499a0f7fecd95f88c15 (127.16.234.131:42843): Network error: Client connection negotiation failed: client connection to 127.16.234.131:42843: connect: Connection refused (error 111)
16:11:56.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:56.671070 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d3c128d101444a59e009013a5c634dc" candidate_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" candidate_term: 10 candidate_status { last_received { term: 9 index: 32 } } ignore_live_leader: false dest_uuid: "d9315253ea9e464c9416773b52ee00a6" is_pre_election: true
16:11:56.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:56.671818 20751 raft_consensus.cc:2454] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bce670edc6c40b2bb1584ec3b364e0f in term 8.
16:11:56.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:56.673144 20309 leader_election.cc:302] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1bce670edc6c40b2bb1584ec3b364e0f, d9315253ea9e464c9416773b52ee00a6; no voters: 3d392ba816c24499a0f7fecd95f88c15
16:11:56.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:56.673949 20850 raft_consensus.cc:2781] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [term 9 FOLLOWER]: Leader pre-election won for term 10
16:11:56.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:56.674211 20850 raft_consensus.cc:482] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [term 9 FOLLOWER]: Starting leader election (detected failure of leader 3d392ba816c24499a0f7fecd95f88c15)
16:11:56.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:56.674552 20850 raft_consensus.cc:3037] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [term 9 FOLLOWER]: Advancing to term 10
16:11:56.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:56.678747 20850 raft_consensus.cc:504] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:11:56.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:56.680095 20850 leader_election.cc:288] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [CANDIDATE]: Term 10 election: Requested vote from peers 3d392ba816c24499a0f7fecd95f88c15 (127.16.234.131:42843), d9315253ea9e464c9416773b52ee00a6 (127.16.234.129:42901)
16:11:56.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:56.681141 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d3c128d101444a59e009013a5c634dc" candidate_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" candidate_term: 10 candidate_status { last_received { term: 9 index: 32 } } ignore_live_leader: false dest_uuid: "d9315253ea9e464c9416773b52ee00a6"
16:11:56.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:56.681735 20751 raft_consensus.cc:3037] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 8 FOLLOWER]: Advancing to term 10
16:11:56.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:56.683034 20303 leader_election.cc:334] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 3d392ba816c24499a0f7fecd95f88c15 (127.16.234.131:42843): Network error: Client connection negotiation failed: client connection to 127.16.234.131:42843: connect: Connection refused (error 111)
16:11:56.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:56.688690 20751 raft_consensus.cc:2454] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bce670edc6c40b2bb1584ec3b364e0f in term 10.
16:11:56.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:56.689424 20309 leader_election.cc:302] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1bce670edc6c40b2bb1584ec3b364e0f, d9315253ea9e464c9416773b52ee00a6; no voters: 3d392ba816c24499a0f7fecd95f88c15
16:11:56.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:56.689986 20850 raft_consensus.cc:2781] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [term 10 FOLLOWER]: Leader election won for term 10
16:11:56.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:56.691431 20850 raft_consensus.cc:686] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [term 10 LEADER]: Becoming Leader. State: Replica: 1bce670edc6c40b2bb1584ec3b364e0f, State: Running, Role: LEADER
16:11:56.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:56.692420 20850 consensus_queue.cc:237] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 9.32, Last appended by leader: 32, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:11:56.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:56.700465 17659 catalog_manager.cc:5495] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f reported cstate change: term changed from 9 to 10, leader changed from 3d392ba816c24499a0f7fecd95f88c15 (127.16.234.131) to 1bce670edc6c40b2bb1584ec3b364e0f (127.16.234.130). New cstate: current_term: 10 leader_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } health_report { overall_health: HEALTHY } } }
16:11:57.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:57.194454 20303 consensus_peers.cc:487] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f -> Peer 3d392ba816c24499a0f7fecd95f88c15 (127.16.234.131:42843): Couldn't send request to peer 3d392ba816c24499a0f7fecd95f88c15. Status: Network error: Client connection negotiation failed: client connection to 127.16.234.131:42843: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
16:11:57.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:57.263391 20751 raft_consensus.cc:1261] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 10 FOLLOWER]: Refusing update from remote peer 1bce670edc6c40b2bb1584ec3b364e0f: Log matching property violated. Preceding OpId in replica: term: 8 index: 28. Preceding OpId from leader: term: 10 index: 33. (index mismatch)
16:11:57.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:57.264828 20850 consensus_queue.cc:1035] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 28, Time since last communication: 0.000s
16:11:58.980 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:11:58.980 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)[L],3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843),d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)]}]
16:11:58.981 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=14, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=3632), Traces: [0ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [3ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [3ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [12ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [13ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [15ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [15ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [32ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [34ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [35ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [35ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [68ms] refreshing cache from master, [69ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [72ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [73ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [75ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [75ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [148ms] refreshing cache from master, [148ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [152ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [154ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [155ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [155ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [309ms] refreshing cache from master, [309ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [312ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [314ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [316ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [316ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [748ms] refreshing cache from master, [748ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [751ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [753ms] sending RPC to server 3d392ba816c24499a0f7fecd95f88c15, [754ms] received response from server 3d392ba816c24499a0f7fecd95f88c15: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [754ms] delaying RPC due to: Network error: Failed to connect to peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843): Connection refused: /127.16.234.131:42843, [3628ms] refreshing cache from master, [3628ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [3631ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@1885593116(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@1cd73cb7 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1988583236 after retry RPC@36662222) -> (continuation of Deferred@803614473 after retry RPC@110988026) -> (continuation of Deferred@573870505 after retry RPC@1209275727) -> (continuation of Deferred@444862449 after retry RPC@1516415466) -> (continuation of Deferred@1041427155 after retry RPC@1911568642) -> (continuation of Deferred@1949091500 after retry RPC@90271915), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@186d863c -> wakeup thread Time-limited test -> (continuation of Deferred@1988583236 after retry RPC@36662222) -> (continuation of Deferred@803614473 after retry RPC@110988026) -> (continuation of Deferred@573870505 after retry RPC@1209275727) -> (continuation of Deferred@444862449 after retry RPC@1516415466) -> (continuation of Deferred@1041427155 after retry RPC@1911568642) -> (continuation of Deferred@1949091500 after retry RPC@90271915))) after lookup
16:11:58.982 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] connecting to peer
16:11:58.983 [DEBUG - kudu-nio-0] (Connection.java:757) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] Successfully connected to peer
16:11:59.004 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0x8d17fe84, L:/127.0.0.1:59638 - R:/127.16.234.130:43937] using TOKEN/null
16:11:59.050 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.16.234.131:42843
16:11:59.051 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

16:11:59.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:59.053332 17322 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu
16:11:59.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu
16:11:59.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-2/wal
16:11:59.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-2/data
16:11:59.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
16:11:59.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
16:11:59.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
16:11:59.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:11:59.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:11:59.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:11:59.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-2/logs
16:11:59.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-2/data/info.pb
16:11:59.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:11:59.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
16:11:59.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
16:11:59.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
16:11:59.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
16:11:59.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:11:59.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:11:59.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:11:59.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
16:11:59.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
16:11:59.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.234.131:42843
16:11:59.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.16.234.131
16:11:59.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.16.234.190:33901,127.16.234.189:32987,127.16.234.188:38065
16:11:59.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=35277
16:11:59.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.234.131
16:11:59.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.234.148:43519
16:11:59.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:11:59.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:11:59.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
16:11:59.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:59.388768 20929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
16:11:59.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:59.389426 20929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
16:11:59.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:59.390205 20929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
16:11:59.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:59.422132 20929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
16:11:59.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:59.423012 20929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.234.131
16:11:59.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:59.457593 20929 tablet_server_runner.cc:78] Tablet server non-default flags:
16:11:59.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:11:59.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.234.148:43519
16:11:59.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:11:59.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
16:11:59.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-2/data
16:11:59.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-2/wal
16:11:59.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:11:59.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:11:59.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.234.131:42843
16:11:59.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
16:11:59.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:11:59.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:11:59.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-2/data/info.pb
16:11:59.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.234.131
16:11:59.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=35277
16:11:59.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.16.234.190:33901,127.16.234.189:32987,127.16.234.188:38065
16:11:59.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
16:11:59.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:11:59.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
16:11:59.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
16:11:59.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:11:59.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.16.234.131
16:11:59.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-2/logs
16:11:59.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:11:59.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
16:11:59.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
16:11:59.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
16:11:59.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
16:11:59.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
16:11:59.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
16:11:59.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 15:12:52 UTC on ccycloud.quasar-tzrfpd.root.comops.site
16:11:59.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4580143
16:11:59.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
16:11:59.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:59.458992 20929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
16:11:59.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:59.460655 20929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
16:11:59.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:59.479874 20943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:11:59.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:59.531342 20303 consensus_peers.cc:487] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f -> Peer 3d392ba816c24499a0f7fecd95f88c15 (127.16.234.131:42843): Couldn't send request to peer 3d392ba816c24499a0f7fecd95f88c15. Status: Network error: Client connection negotiation failed: client connection to 127.16.234.131:42843: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
16:12:00.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:59.479887 20942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:12:00.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:11:59.481009 20929 server_base.cc:1034] running on GCE node
16:12:00.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:11:59.480968 20946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:12:00.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:00.599172 20929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
16:12:00.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:00.601843 20929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
16:12:00.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:00.603298 20929 hybrid_clock.cc:648] HybridClock initialized: now 1786464720603238 us; error 75 us; skew 500 ppm
16:12:00.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:00.604162 20929 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
16:12:00.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:00.615362 20929 webserver.cc:460] Webserver started at http://127.16.234.131:35277/ using document root <none> and password file <none>
16:12:00.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:00.616345 20929 fs_manager.cc:359] Metadata directory not provided
16:12:00.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:00.616601 20929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
16:12:00.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:00.624413 20929 fs_manager.cc:709] Time spent opening directory manager: real 0.005s	user 0.003s	sys 0.002s
16:12:00.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:00.628930 20961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
16:12:00.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:00.629940 20929 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
16:12:00.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:00.630307 20929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-2/data,/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-2/wal
16:12:00.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "3d392ba816c24499a0f7fecd95f88c15"
16:12:00.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 16:11:18 on dist-test-slave-zj8q"
16:12:00.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:00.632750 20929 fs_report.cc:383] FS layout report
16:12:00.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
16:12:00.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-2/wal
16:12:00.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-2/wal
16:12:00.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-2/data/data
16:12:00.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
16:12:00.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
16:12:00.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
16:12:00.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
16:12:00.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
16:12:00.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
16:12:00.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
16:12:00.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
16:12:00.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
16:12:00.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
16:12:00.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
16:12:00.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:00.671348 20929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
16:12:00.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:00.673065 20929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
16:12:00.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:00.673574 20929 kserver.cc:163] Server-wide thread pool size limit: 3276
16:12:00.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:00.676258 20929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
16:12:00.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:00.682590 20975 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
16:12:00.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:00.690282 20929 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
16:12:00.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:00.690539 20929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s	user 0.000s	sys 0.002s
16:12:00.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:00.690903 20929 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
16:12:00.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:00.695776 20929 ts_tablet_manager.cc:610] Registered 1 tablets
16:12:00.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:00.695958 20929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s	user 0.003s	sys 0.001s
16:12:00.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:00.696283 20975 tablet_bootstrap.cc:492] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15: Bootstrap starting.
16:12:00.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:00.758831 20975 log.cc:824] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15: Log is configured to *not* fsync() on all Append() calls
16:12:00.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:00.862635 20929 rpc_server.cc:307] RPC server started. Bound to: 127.16.234.131:42843
16:12:00.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:00.862813 21168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.234.131:42843 every 8 connection(s)
16:12:00.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:00.866259 20929 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-2/data/info.pb
16:12:00.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:00.868743 17322 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu as pid 20929
16:12:00.869 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
16:12:00.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:00.874065 20975 tablet_bootstrap.cc:492] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=24 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
16:12:00.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:00.875921 20975 tablet_bootstrap.cc:492] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15: Bootstrap complete.
16:12:00.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:00.877640 20975 ts_tablet_manager.cc:1397] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15: Time spent bootstrapping tablet: real 0.182s	user 0.155s	sys 0.022s
16:12:00.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:00.906100 20975 raft_consensus.cc:348] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 9 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:12:00.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:00.909382 20975 raft_consensus.cc:729] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 9 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d392ba816c24499a0f7fecd95f88c15, State: Initialized, Role: FOLLOWER
16:12:00.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:00.918906 20975 consensus_queue.cc:260] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 9.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:12:00.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:00.932899 20975 ts_tablet_manager.cc:1428] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15: Time spent starting tablet: real 0.055s	user 0.023s	sys 0.011s
16:12:00.938 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 27 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304, tablet=7d3c128d101444a59e009013a5c634dc, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
16:12:00.938 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.16.234.130:43937
16:12:00.939 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

16:12:00.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:00.939416 17322 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu with pid 20268
16:12:00.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:00.955564 21169 heartbeater.cc:347] Connected to a master server at 127.16.234.188:38065
16:12:00.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:00.956051 21169 heartbeater.cc:464] Registering TS with master...
16:12:00.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:00.957294 21169 heartbeater.cc:511] Master 127.16.234.188:38065 requested a full tablet report, sending...
16:12:00.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:00.962544 17662 ts_manager.cc:194] Re-registered known tserver with Master: 3d392ba816c24499a0f7fecd95f88c15 (127.16.234.131:42843)
16:12:00.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:00.967303 17662 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.234.131:36543
16:12:00.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:00.968377 21177 heartbeater.cc:347] Connected to a master server at 127.16.234.190:33901
16:12:00.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:00.968760 21177 heartbeater.cc:464] Registering TS with master...
16:12:00.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:00.969647 21177 heartbeater.cc:511] Master 127.16.234.190:33901 requested a full tablet report, sending...
16:12:00.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:00.973147 21169 heartbeater.cc:503] Master 127.16.234.188:38065 was elected leader, sending a full tablet report...
16:12:00.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:00.973922 17402 ts_manager.cc:194] Re-registered known tserver with Master: 3d392ba816c24499a0f7fecd95f88c15 (127.16.234.131:42843)
16:12:00.984 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:00.984380 21170 heartbeater.cc:347] Connected to a master server at 127.16.234.189:32987
16:12:00.984 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:00.984740 21170 heartbeater.cc:464] Registering TS with master...
16:12:00.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:00.985445 21170 heartbeater.cc:511] Master 127.16.234.189:32987 requested a full tablet report, sending...
16:12:00.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:00.990315 17536 ts_manager.cc:194] Re-registered known tserver with Master: 3d392ba816c24499a0f7fecd95f88c15 (127.16.234.131:42843)
16:12:00.992 [DEBUG - kudu-nio-0] (Connection.java:239) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] handling channelInactive
16:12:00.992 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] cleaning up while in state READY due to: connection closed
16:12:00.992 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
16:12:00.993 [DEBUG - Time-limited test] (Connection.java:748) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] connecting to peer
16:12:00.995 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:12:00.995 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:12:00.995 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:12:01.001 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:12:01.001 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)[L],3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843),d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)]}]
16:12:01.001 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=2, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=9), Traces: [0ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [3ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [3ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [8ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@1020886118(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@408da3b5 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@73d0f402 -> wakeup thread Time-limited test)) after lookup
16:12:01.002 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] connecting to peer
16:12:01.004 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:12:01.004 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:12:01.004 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:12:01.020 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:12:01.020 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)[L],3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843),d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)]}]
16:12:01.021 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=4, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=28), Traces: [0ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [3ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [3ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [8ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [10ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [12ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [12ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [27ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@1020886118(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@408da3b5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1116512052 after retry RPC@906498947), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@73d0f402 -> wakeup thread Time-limited test -> (continuation of Deferred@1116512052 after retry RPC@906498947))) after lookup
16:12:01.021 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] connecting to peer
16:12:01.022 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:12:01.022 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:12:01.022 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:12:01.060 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:12:01.060 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)[L],3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843),d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)]}]
16:12:01.061 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=6, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=68), Traces: [0ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [3ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [3ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [8ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [10ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [12ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [12ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [27ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [29ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [30ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [30ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [68ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@1020886118(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@408da3b5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1116512052 after retry RPC@906498947) -> (continuation of Deferred@1517664510 after retry RPC@1896018602), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@73d0f402 -> wakeup thread Time-limited test -> (continuation of Deferred@1116512052 after retry RPC@906498947) -> (continuation of Deferred@1517664510 after retry RPC@1896018602))) after lookup
16:12:01.061 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] connecting to peer
16:12:01.062 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:12:01.062 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:12:01.062 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:12:01.080 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:12:01.080 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)[L],3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843),d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)]}]
16:12:01.081 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=8, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=88), Traces: [0ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [3ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [3ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [8ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [10ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [12ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [12ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [27ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [29ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [30ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [30ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [68ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [69ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [70ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [70ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [84ms] refreshing cache from master, [85ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [88ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@1020886118(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@408da3b5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1116512052 after retry RPC@906498947) -> (continuation of Deferred@1517664510 after retry RPC@1896018602) -> (continuation of Deferred@1681384684 after retry RPC@46587234), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@73d0f402 -> wakeup thread Time-limited test -> (continuation of Deferred@1116512052 after retry RPC@906498947) -> (continuation of Deferred@1517664510 after retry RPC@1896018602) -> (continuation of Deferred@1681384684 after retry RPC@46587234))) after lookup
16:12:01.081 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] connecting to peer
16:12:01.082 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:12:01.082 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:12:01.082 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:12:01.400 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:12:01.401 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)[L],3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843),d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)]}]
16:12:01.401 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=10, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=408), Traces: [0ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [3ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [3ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [8ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [10ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [12ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [12ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [27ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [29ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [30ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [30ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [68ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [69ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [70ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [70ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [84ms] refreshing cache from master, [85ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [88ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [89ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [90ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [90ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [405ms] refreshing cache from master, [405ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [408ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@1020886118(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@408da3b5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1116512052 after retry RPC@906498947) -> (continuation of Deferred@1517664510 after retry RPC@1896018602) -> (continuation of Deferred@1681384684 after retry RPC@46587234) -> (continuation of Deferred@449368101 after retry RPC@160541124), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@73d0f402 -> wakeup thread Time-limited test -> (continuation of Deferred@1116512052 after retry RPC@906498947) -> (continuation of Deferred@1517664510 after retry RPC@1896018602) -> (continuation of Deferred@1681384684 after retry RPC@46587234) -> (continuation of Deferred@449368101 after retry RPC@160541124))) after lookup
16:12:01.402 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] connecting to peer
16:12:01.403 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:12:01.403 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:12:01.403 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937
16:12:01.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:01.993384 21197 raft_consensus.cc:482] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 1bce670edc6c40b2bb1584ec3b364e0f)
16:12:01.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:01.993819 21197 raft_consensus.cc:504] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:12:01.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:01.995221 21197 leader_election.cc:288] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3d392ba816c24499a0f7fecd95f88c15 (127.16.234.131:42843), 1bce670edc6c40b2bb1584ec3b364e0f (127.16.234.130:43937)
16:12:02.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:12:01.999751 20630 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.234.130:43937: connect: Connection refused (error 111)
16:12:02.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:12:02.003304 20630 leader_election.cc:334] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 1bce670edc6c40b2bb1584ec3b364e0f (127.16.234.130:43937): Network error: Client connection negotiation failed: client connection to 127.16.234.130:43937: connect: Connection refused (error 111)
16:12:02.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:02.012413 21092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d3c128d101444a59e009013a5c634dc" candidate_uuid: "d9315253ea9e464c9416773b52ee00a6" candidate_term: 11 candidate_status { last_received { term: 10 index: 36 } } ignore_live_leader: false dest_uuid: "3d392ba816c24499a0f7fecd95f88c15" is_pre_election: true
16:12:02.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:02.013173 21092 raft_consensus.cc:2454] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9315253ea9e464c9416773b52ee00a6 in term 9.
16:12:02.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:02.014396 20627 leader_election.cc:302] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d392ba816c24499a0f7fecd95f88c15, d9315253ea9e464c9416773b52ee00a6; no voters: 1bce670edc6c40b2bb1584ec3b364e0f
16:12:02.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:02.015179 21197 raft_consensus.cc:2781] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 10 FOLLOWER]: Leader pre-election won for term 11
16:12:02.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:02.015496 21197 raft_consensus.cc:482] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 1bce670edc6c40b2bb1584ec3b364e0f)
16:12:02.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:02.015792 21197 raft_consensus.cc:3037] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 10 FOLLOWER]: Advancing to term 11
16:12:02.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:02.022078 21197 raft_consensus.cc:504] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:12:02.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:02.024778 21197 leader_election.cc:288] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [CANDIDATE]: Term 11 election: Requested vote from peers 3d392ba816c24499a0f7fecd95f88c15 (127.16.234.131:42843), 1bce670edc6c40b2bb1584ec3b364e0f (127.16.234.130:43937)
16:12:02.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:02.024780 21092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d3c128d101444a59e009013a5c634dc" candidate_uuid: "d9315253ea9e464c9416773b52ee00a6" candidate_term: 11 candidate_status { last_received { term: 10 index: 36 } } ignore_live_leader: false dest_uuid: "3d392ba816c24499a0f7fecd95f88c15"
16:12:02.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:02.025353 21092 raft_consensus.cc:3037] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 9 FOLLOWER]: Advancing to term 11
16:12:02.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:12:02.029244 20630 leader_election.cc:334] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 1bce670edc6c40b2bb1584ec3b364e0f (127.16.234.130:43937): Network error: Client connection negotiation failed: client connection to 127.16.234.130:43937: connect: Connection refused (error 111)
16:12:02.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:02.034297 21092 raft_consensus.cc:2454] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9315253ea9e464c9416773b52ee00a6 in term 11.
16:12:02.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:02.035351 20627 leader_election.cc:302] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d392ba816c24499a0f7fecd95f88c15, d9315253ea9e464c9416773b52ee00a6; no voters: 1bce670edc6c40b2bb1584ec3b364e0f
16:12:02.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:02.035980 21197 raft_consensus.cc:2781] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 11 FOLLOWER]: Leader election won for term 11
16:12:02.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:02.037014 21197 raft_consensus.cc:686] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 11 LEADER]: Becoming Leader. State: Replica: d9315253ea9e464c9416773b52ee00a6, State: Running, Role: LEADER
16:12:02.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:02.037779 21197 consensus_queue.cc:237] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 10.36, Last appended by leader: 36, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:12:02.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:02.047825 17662 catalog_manager.cc:5495] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 reported cstate change: term changed from 10 to 11, leader changed from 1bce670edc6c40b2bb1584ec3b364e0f (127.16.234.130) to d9315253ea9e464c9416773b52ee00a6 (127.16.234.129). New cstate: current_term: 11 leader_uuid: "d9315253ea9e464c9416773b52ee00a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } health_report { overall_health: UNKNOWN } } }
16:12:02.420 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:12:02.421 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937),3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843),d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)[L]]}]
16:12:02.421 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=12, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=1428), Traces: [0ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [3ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [3ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [8ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [10ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [12ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [12ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [27ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [29ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [30ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [30ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [68ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [69ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [70ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [70ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [84ms] refreshing cache from master, [85ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [88ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [89ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [90ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [90ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [405ms] refreshing cache from master, [405ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [408ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [410ms] sending RPC to server 1bce670edc6c40b2bb1584ec3b364e0f, [411ms] received response from server 1bce670edc6c40b2bb1584ec3b364e0f: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [412ms] delaying RPC due to: Network error: Failed to connect to peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937): Connection refused: /127.16.234.130:43937, [1425ms] refreshing cache from master, [1425ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [1428ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@1020886118(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@408da3b5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1116512052 after retry RPC@906498947) -> (continuation of Deferred@1517664510 after retry RPC@1896018602) -> (continuation of Deferred@1681384684 after retry RPC@46587234) -> (continuation of Deferred@449368101 after retry RPC@160541124) -> (continuation of Deferred@777824394 after retry RPC@713550604), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@73d0f402 -> wakeup thread Time-limited test -> (continuation of Deferred@1116512052 after retry RPC@906498947) -> (continuation of Deferred@1517664510 after retry RPC@1896018602) -> (continuation of Deferred@1681384684 after retry RPC@46587234) -> (continuation of Deferred@449368101 after retry RPC@160541124) -> (continuation of Deferred@777824394 after retry RPC@713550604))) after lookup
16:12:02.422 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)] connecting to peer
16:12:02.423 [DEBUG - kudu-nio-1] (Connection.java:757) [peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)] Successfully connected to peer
16:12:02.443 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x772d29dd, L:/127.0.0.1:42534 - R:/127.16.234.129:42901] using TOKEN/null
16:12:02.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:02.453513 21092 raft_consensus.cc:1261] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 11 FOLLOWER]: Refusing update from remote peer d9315253ea9e464c9416773b52ee00a6: Log matching property violated. Preceding OpId in replica: term: 9 index: 32. Preceding OpId from leader: term: 11 index: 38. (index mismatch)
16:12:02.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:02.455154 21203 consensus_queue.cc:1035] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 32, Time since last communication: 0.001s
16:12:02.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:12:02.456259 20630 consensus_peers.cc:487] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 -> Peer 1bce670edc6c40b2bb1584ec3b364e0f (127.16.234.130:43937): Couldn't send request to peer 1bce670edc6c40b2bb1584ec3b364e0f. Status: Network error: Client connection negotiation failed: client connection to 127.16.234.130:43937: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
16:12:02.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:02.471086 21232 mvcc.cc:205] Tried to move back new op lower bound from 7317359489062772736 to 7317359479579168768. Current Snapshot: MvccSnapshot[applied={T|T < 7317359489062772736}]
16:12:02.518 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.16.234.130:43937
16:12:02.518 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

16:12:02.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:02.519845 17322 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu
16:12:02.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu
16:12:02.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/wal
16:12:02.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/data
16:12:02.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
16:12:02.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
16:12:02.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
16:12:02.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:12:02.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:12:02.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:12:02.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/logs
16:12:02.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/data/info.pb
16:12:02.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:12:02.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
16:12:02.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
16:12:02.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
16:12:02.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
16:12:02.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:12:02.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:12:02.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:12:02.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
16:12:02.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
16:12:02.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.234.130:43937
16:12:02.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.16.234.130
16:12:02.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.16.234.190:33901,127.16.234.189:32987,127.16.234.188:38065
16:12:02.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=37863
16:12:02.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.234.130
16:12:02.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.234.148:43519
16:12:02.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:12:02.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:12:02.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
16:12:02.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:12:02.996691 21242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
16:12:02.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:12:02.997406 21242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
16:12:02.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:12:02.997887 21242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
16:12:03.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:12:03.031751 21242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
16:12:03.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:12:03.032563 21242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.234.130
16:12:03.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:03.069012 21242 tablet_server_runner.cc:78] Tablet server non-default flags:
16:12:03.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:12:03.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.234.148:43519
16:12:03.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:12:03.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
16:12:03.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/data
16:12:03.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/wal
16:12:03.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:12:03.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:12:03.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.234.130:43937
16:12:03.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
16:12:03.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:12:03.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:12:03.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/data/info.pb
16:12:03.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.234.130
16:12:03.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=37863
16:12:03.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.16.234.190:33901,127.16.234.189:32987,127.16.234.188:38065
16:12:03.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
16:12:03.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:12:03.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
16:12:03.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
16:12:03.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:12:03.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.16.234.130
16:12:03.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/logs
16:12:03.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:12:03.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
16:12:03.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
16:12:03.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
16:12:03.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
16:12:03.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
16:12:03.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
16:12:03.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 15:12:52 UTC on ccycloud.quasar-tzrfpd.root.comops.site
16:12:03.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4580143
16:12:03.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
16:12:03.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:03.070492 21242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
16:12:03.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:03.072065 21242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
16:12:03.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:12:03.111270 21254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:12:04.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:12:03.111461 21256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:12:04.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:12:03.114140 21259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:12:04.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:03.117458 21242 server_base.cc:1034] running on GCE node
16:12:04.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:04.231807 21242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
16:12:04.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:04.234582 21242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
16:12:04.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:04.236006 21242 hybrid_clock.cc:648] HybridClock initialized: now 1786464724235971 us; error 50 us; skew 500 ppm
16:12:04.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:04.236792 21242 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
16:12:04.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:04.244751 21242 webserver.cc:460] Webserver started at http://127.16.234.130:37863/ using document root <none> and password file <none>
16:12:04.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:04.245788 21242 fs_manager.cc:359] Metadata directory not provided
16:12:04.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:04.246021 21242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
16:12:04.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:04.254444 21242 fs_manager.cc:709] Time spent opening directory manager: real 0.005s	user 0.006s	sys 0.001s
16:12:04.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:04.259102 21277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
16:12:04.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:04.260195 21242 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.002s
16:12:04.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:04.260571 21242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/data,/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/wal
16:12:04.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "1bce670edc6c40b2bb1584ec3b364e0f"
16:12:04.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 16:11:16 on dist-test-slave-zj8q"
16:12:04.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:04.263059 21242 fs_report.cc:383] FS layout report
16:12:04.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
16:12:04.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/wal
16:12:04.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/wal
16:12:04.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/data/data
16:12:04.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
16:12:04.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
16:12:04.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
16:12:04.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
16:12:04.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
16:12:04.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
16:12:04.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
16:12:04.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
16:12:04.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
16:12:04.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
16:12:04.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
16:12:04.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:04.297940 21242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
16:12:04.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:04.299547 21242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
16:12:04.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:04.300058 21242 kserver.cc:163] Server-wide thread pool size limit: 3276
16:12:04.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:04.302721 21242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
16:12:04.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:04.308561 21287 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
16:12:04.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:04.316771 21242 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
16:12:04.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:04.317018 21242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s	user 0.001s	sys 0.001s
16:12:04.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:04.317364 21242 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
16:12:04.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:04.322283 21242 ts_tablet_manager.cc:610] Registered 1 tablets
16:12:04.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:04.322468 21242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s	user 0.005s	sys 0.000s
16:12:04.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:04.322836 21287 tablet_bootstrap.cc:492] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f: Bootstrap starting.
16:12:04.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:04.397056 21287 log.cc:824] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f: Log is configured to *not* fsync() on all Append() calls
16:12:04.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:04.492199 21242 rpc_server.cc:307] RPC server started. Bound to: 127.16.234.130:43937
16:12:04.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:04.492437 21455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.234.130:43937 every 8 connection(s)
16:12:04.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:04.495080 21242 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-1/data/info.pb
16:12:04.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:04.496707 17322 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu as pid 21242
16:12:04.497 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
16:12:04.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:04.537832 21287 tablet_bootstrap.cc:492] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f: Bootstrap replayed 1/1 log segments. Stats: ops{read=36 overwritten=0 applied=36 ignored=0} inserts{seen=27 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
16:12:04.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:04.539178 21287 tablet_bootstrap.cc:492] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f: Bootstrap complete.
16:12:04.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:04.541203 21287 ts_tablet_manager.cc:1397] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f: Time spent bootstrapping tablet: real 0.219s	user 0.144s	sys 0.045s
16:12:04.551 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 30 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304, tablet=7d3c128d101444a59e009013a5c634dc, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
16:12:04.551 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.16.234.129:42901
16:12:04.552 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

16:12:04.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:04.552604 17322 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu with pid 20592
16:12:04.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:04.575932 21287 raft_consensus.cc:348] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [term 10 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:12:04.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:04.581127 21287 raft_consensus.cc:729] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [term 10 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bce670edc6c40b2bb1584ec3b364e0f, State: Initialized, Role: FOLLOWER
16:12:04.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:04.585731 21287 consensus_queue.cc:260] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 36, Last appended: 10.36, Last appended by leader: 36, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:12:04.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:04.592628 21457 heartbeater.cc:347] Connected to a master server at 127.16.234.188:38065
16:12:04.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:04.598089 21457 heartbeater.cc:464] Registering TS with master...
16:12:04.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:04.599320 21457 heartbeater.cc:511] Master 127.16.234.188:38065 requested a full tablet report, sending...
16:12:04.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:04.604102 17662 ts_manager.cc:194] Re-registered known tserver with Master: 1bce670edc6c40b2bb1584ec3b364e0f (127.16.234.130:43937)
16:12:04.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:04.607627 17662 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.234.130:41077
16:12:04.614 [DEBUG - kudu-nio-1] (Connection.java:239) [peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)] handling channelInactive
16:12:04.614 [DEBUG - kudu-nio-1] (Connection.java:709) [peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)] cleaning up while in state READY due to: connection closed
16:12:04.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:04.614943 21287 ts_tablet_manager.cc:1428] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f: Time spent starting tablet: real 0.073s	user 0.029s	sys 0.005s
16:12:04.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:12:04.615502 21466 negotiation.cc:337] Failed RPC negotiation. Trace:
16:12:04.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:12:04.542650 (+     0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.234.129:33659 (local address 127.16.234.130:43937)
16:12:04.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:12:04.551213 (+  8563us) server_negotiation.cc:206] Beginning negotiation
16:12:04.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:12:04.551254 (+    41us) server_negotiation.cc:399] Waiting for connection header
16:12:04.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:12:04.551339 (+    85us) server_negotiation.cc:407] Connection header received
16:12:04.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:12:04.573273 (+ 21934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
16:12:04.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:12:04.573330 (+    57us) server_negotiation.cc:461] Received NEGOTIATE request from client
16:12:04.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:12:04.573811 (+   481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
16:12:04.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:12:04.574586 (+   775us) server_negotiation.cc:226] Negotiated authn=SASL
16:12:04.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:12:04.613740 (+ 39154us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.234.129:33659: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
16:12:04.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":7906,"thread_start_us":8960,"threads_started":1}
16:12:04.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:04.616315 21457 heartbeater.cc:503] Master 127.16.234.188:38065 was elected leader, sending a full tablet report...
16:12:04.618 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
16:12:04.619 [DEBUG - Time-limited test] (Connection.java:748) [peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)] connecting to peer
16:12:04.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:04.619056 21459 heartbeater.cc:347] Connected to a master server at 127.16.234.189:32987
16:12:04.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:04.619390 21459 heartbeater.cc:464] Registering TS with master...
16:12:04.620 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901
16:12:04.620 [DEBUG - kudu-nio-0] (Connection.java:709) [peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)] cleaning up while in state CONNECTING due to: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901
16:12:04.620 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901
16:12:04.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:04.622188 21459 heartbeater.cc:511] Master 127.16.234.189:32987 requested a full tablet report, sending...
16:12:04.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:04.626786 17536 ts_manager.cc:194] Re-registered known tserver with Master: 1bce670edc6c40b2bb1584ec3b364e0f (127.16.234.130:43937)
16:12:04.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:04.629894 21468 heartbeater.cc:347] Connected to a master server at 127.16.234.190:33901
16:12:04.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:04.630216 21468 heartbeater.cc:464] Registering TS with master...
16:12:04.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:04.630798 21468 heartbeater.cc:511] Master 127.16.234.190:33901 requested a full tablet report, sending...
16:12:04.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:04.633572 17401 ts_manager.cc:194] Re-registered known tserver with Master: 1bce670edc6c40b2bb1584ec3b364e0f (127.16.234.130:43937)
16:12:04.640 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:12:04.641 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937),3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843),d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)[L]]}]
16:12:04.641 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=2, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=22), Traces: [0ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [2ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [2ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [19ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [22ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@735538554(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@23a92d34 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6c2291b8 -> wakeup thread Time-limited test)) after lookup
16:12:04.641 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)] connecting to peer
16:12:04.643 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901
16:12:04.643 [DEBUG - kudu-nio-1] (Connection.java:709) [peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)] cleaning up while in state CONNECTING due to: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901
16:12:04.643 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901
16:12:04.659 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:12:04.660 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937),3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843),d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)[L]]}]
16:12:04.660 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=4, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=41), Traces: [0ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [2ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [2ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [19ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [22ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [23ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [25ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [25ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [41ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@735538554(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@23a92d34 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@219321008 after retry RPC@975893672), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6c2291b8 -> wakeup thread Time-limited test -> (continuation of Deferred@219321008 after retry RPC@975893672))) after lookup
16:12:04.661 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)] connecting to peer
16:12:04.662 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901
16:12:04.662 [DEBUG - kudu-nio-0] (Connection.java:709) [peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)] cleaning up while in state CONNECTING due to: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901
16:12:04.662 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901
16:12:04.701 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:12:04.701 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937),3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843),d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)[L]]}]
16:12:04.702 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=6, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=83), Traces: [0ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [2ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [2ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [19ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [22ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [23ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [25ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [25ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [41ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [43ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [44ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [44ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [79ms] refreshing cache from master, [79ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [82ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@735538554(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@23a92d34 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@219321008 after retry RPC@975893672) -> (continuation of Deferred@1684123059 after retry RPC@159619619), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6c2291b8 -> wakeup thread Time-limited test -> (continuation of Deferred@219321008 after retry RPC@975893672) -> (continuation of Deferred@1684123059 after retry RPC@159619619))) after lookup
16:12:04.702 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)] connecting to peer
16:12:04.703 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901
16:12:04.703 [DEBUG - kudu-nio-1] (Connection.java:709) [peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)] cleaning up while in state CONNECTING due to: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901
16:12:04.703 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901
16:12:04.740 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:12:04.741 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937),3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843),d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)[L]]}]
16:12:04.741 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=8, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=122), Traces: [0ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [2ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [2ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [19ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [22ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [23ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [25ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [25ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [41ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [43ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [44ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [44ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [79ms] refreshing cache from master, [79ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [82ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [84ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [85ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [85ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [118ms] refreshing cache from master, [119ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [122ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@735538554(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@23a92d34 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@219321008 after retry RPC@975893672) -> (continuation of Deferred@1684123059 after retry RPC@159619619) -> (continuation of Deferred@1549874624 after retry RPC@1516962747), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6c2291b8 -> wakeup thread Time-limited test -> (continuation of Deferred@219321008 after retry RPC@975893672) -> (continuation of Deferred@1684123059 after retry RPC@159619619) -> (continuation of Deferred@1549874624 after retry RPC@1516962747))) after lookup
16:12:04.742 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)] connecting to peer
16:12:04.743 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901
16:12:04.743 [DEBUG - kudu-nio-0] (Connection.java:709) [peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)] cleaning up while in state CONNECTING due to: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901
16:12:04.743 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901
16:12:05.240 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:12:05.240 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937),3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843),d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)[L]]}]
16:12:05.240 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=10, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=622), Traces: [0ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [2ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [2ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [19ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [22ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [23ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [25ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [25ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [41ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [43ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [44ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [44ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [79ms] refreshing cache from master, [79ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [82ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [84ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [85ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [85ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [118ms] refreshing cache from master, [119ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [122ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [123ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [125ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [125ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [618ms] refreshing cache from master, [618ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [621ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@735538554(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@23a92d34 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@219321008 after retry RPC@975893672) -> (continuation of Deferred@1684123059 after retry RPC@159619619) -> (continuation of Deferred@1549874624 after retry RPC@1516962747) -> (continuation of Deferred@618881349 after retry RPC@639092921), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6c2291b8 -> wakeup thread Time-limited test -> (continuation of Deferred@219321008 after retry RPC@975893672) -> (continuation of Deferred@1684123059 after retry RPC@159619619) -> (continuation of Deferred@1549874624 after retry RPC@1516962747) -> (continuation of Deferred@618881349 after retry RPC@639092921))) after lookup
16:12:05.241 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)] connecting to peer
16:12:05.242 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901
16:12:05.242 [DEBUG - kudu-nio-1] (Connection.java:709) [peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)] cleaning up while in state CONNECTING due to: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901
16:12:05.242 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901) for tablet 7d3c128d101444a59e009013a5c634dc: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901
16:12:05.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:05.694481 21479 raft_consensus.cc:482] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader d9315253ea9e464c9416773b52ee00a6)
16:12:05.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:05.694890 21479 raft_consensus.cc:504] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:12:05.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:05.697160 21479 leader_election.cc:288] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers d9315253ea9e464c9416773b52ee00a6 (127.16.234.129:42901), 1bce670edc6c40b2bb1584ec3b364e0f (127.16.234.130:43937)
16:12:05.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:12:05.700184 20968 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.234.129:42901: connect: Connection refused (error 111)
16:12:05.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:12:05.703715 20968 leader_election.cc:334] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer d9315253ea9e464c9416773b52ee00a6 (127.16.234.129:42901): Network error: Client connection negotiation failed: client connection to 127.16.234.129:42901: connect: Connection refused (error 111)
16:12:05.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:05.711257 21381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d3c128d101444a59e009013a5c634dc" candidate_uuid: "3d392ba816c24499a0f7fecd95f88c15" candidate_term: 12 candidate_status { last_received { term: 11 index: 40 } } ignore_live_leader: false dest_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" is_pre_election: true
16:12:05.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:05.711922 21381 raft_consensus.cc:2454] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d392ba816c24499a0f7fecd95f88c15 in term 10.
16:12:05.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:05.713160 20969 leader_election.cc:302] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1bce670edc6c40b2bb1584ec3b364e0f, 3d392ba816c24499a0f7fecd95f88c15; no voters: d9315253ea9e464c9416773b52ee00a6
16:12:05.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:05.713804 21479 raft_consensus.cc:2781] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 11 FOLLOWER]: Leader pre-election won for term 12
16:12:05.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:05.714040 21479 raft_consensus.cc:482] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 11 FOLLOWER]: Starting leader election (detected failure of leader d9315253ea9e464c9416773b52ee00a6)
16:12:05.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:05.714284 21479 raft_consensus.cc:3037] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 11 FOLLOWER]: Advancing to term 12
16:12:05.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:05.720577 21479 raft_consensus.cc:504] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:12:05.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:05.722477 21479 leader_election.cc:288] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [CANDIDATE]: Term 12 election: Requested vote from peers d9315253ea9e464c9416773b52ee00a6 (127.16.234.129:42901), 1bce670edc6c40b2bb1584ec3b364e0f (127.16.234.130:43937)
16:12:05.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:05.723627 21381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d3c128d101444a59e009013a5c634dc" candidate_uuid: "3d392ba816c24499a0f7fecd95f88c15" candidate_term: 12 candidate_status { last_received { term: 11 index: 40 } } ignore_live_leader: false dest_uuid: "1bce670edc6c40b2bb1584ec3b364e0f"
16:12:05.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:05.724190 21381 raft_consensus.cc:3037] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [term 10 FOLLOWER]: Advancing to term 12
16:12:05.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:12:05.725796 20968 leader_election.cc:334] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer d9315253ea9e464c9416773b52ee00a6 (127.16.234.129:42901): Network error: Client connection negotiation failed: client connection to 127.16.234.129:42901: connect: Connection refused (error 111)
16:12:05.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:05.731017 21381 raft_consensus.cc:2454] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d392ba816c24499a0f7fecd95f88c15 in term 12.
16:12:05.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:05.731891 20969 leader_election.cc:302] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1bce670edc6c40b2bb1584ec3b364e0f, 3d392ba816c24499a0f7fecd95f88c15; no voters: d9315253ea9e464c9416773b52ee00a6
16:12:05.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:05.732474 21479 raft_consensus.cc:2781] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 12 FOLLOWER]: Leader election won for term 12
16:12:05.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:05.733932 21479 raft_consensus.cc:686] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [term 12 LEADER]: Becoming Leader. State: Replica: 3d392ba816c24499a0f7fecd95f88c15, State: Running, Role: LEADER
16:12:05.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:05.734776 21479 consensus_queue.cc:237] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 11.40, Last appended by leader: 40, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:12:05.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:05.742871 17662 catalog_manager.cc:5495] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 reported cstate change: term changed from 11 to 12, leader changed from d9315253ea9e464c9416773b52ee00a6 (127.16.234.129) to 3d392ba816c24499a0f7fecd95f88c15 (127.16.234.131). New cstate: current_term: 12 leader_uuid: "3d392ba816c24499a0f7fecd95f88c15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } health_report { overall_health: UNKNOWN } } }
16:12:06.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:12:06.277221 20968 consensus_peers.cc:487] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 -> Peer d9315253ea9e464c9416773b52ee00a6 (127.16.234.129:42901): Couldn't send request to peer d9315253ea9e464c9416773b52ee00a6. Status: Network error: Client connection negotiation failed: client connection to 127.16.234.129:42901: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
16:12:06.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:06.328071 21381 raft_consensus.cc:1261] T 7d3c128d101444a59e009013a5c634dc P 1bce670edc6c40b2bb1584ec3b364e0f [term 12 FOLLOWER]: Refusing update from remote peer 3d392ba816c24499a0f7fecd95f88c15: Log matching property violated. Preceding OpId in replica: term: 10 index: 36. Preceding OpId from leader: term: 12 index: 41. (index mismatch)
16:12:06.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:06.329445 21500 consensus_queue.cc:1035] T 7d3c128d101444a59e009013a5c634dc P 3d392ba816c24499a0f7fecd95f88c15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 36, Time since last communication: 0.000s
16:12:06.640 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 7d3c128d101444a59e009013a5c634dc for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304' with partition [<start>, <end>)
16:12:06.641 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7d3c128d101444a59e009013a5c634dc@[1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937),3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)[L],d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)]}]
16:12:06.641 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7d3c128d101444a59e009013a5c634dc, attempt=12, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=2022), Traces: [0ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [2ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [2ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [19ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [22ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [23ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [25ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [25ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [41ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [43ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [44ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [44ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [79ms] refreshing cache from master, [79ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [82ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [84ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [85ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [85ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [118ms] refreshing cache from master, [119ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [122ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [123ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [125ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [125ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [618ms] refreshing cache from master, [618ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [621ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, [623ms] sending RPC to server d9315253ea9e464c9416773b52ee00a6, [624ms] received response from server d9315253ea9e464c9416773b52ee00a6: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [624ms] delaying RPC due to: Network error: Failed to connect to peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901): Connection refused: /127.16.234.129:42901, [2018ms] refreshing cache from master, [2018ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.234.188:38065, [2022ms] Sub RPC GetTableLocations: received response from server master-127.16.234.188:38065: OK, deferred=Deferred@735538554(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@23a92d34 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@219321008 after retry RPC@975893672) -> (continuation of Deferred@1684123059 after retry RPC@159619619) -> (continuation of Deferred@1549874624 after retry RPC@1516962747) -> (continuation of Deferred@618881349 after retry RPC@639092921) -> (continuation of Deferred@1183607786 after retry RPC@211964674), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6c2291b8 -> wakeup thread Time-limited test -> (continuation of Deferred@219321008 after retry RPC@975893672) -> (continuation of Deferred@1684123059 after retry RPC@159619619) -> (continuation of Deferred@1549874624 after retry RPC@1516962747) -> (continuation of Deferred@618881349 after retry RPC@639092921) -> (continuation of Deferred@1183607786 after retry RPC@211964674))) after lookup
16:12:06.642 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)] connecting to peer
16:12:06.643 [DEBUG - kudu-nio-0] (Connection.java:757) [peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)] Successfully connected to peer
16:12:06.662 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0xf9ea66e9, L:/127.0.0.1:42802 - R:/127.16.234.131:42843] using TOKEN/null
16:12:06.702 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.16.234.129:42901
16:12:06.702 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

16:12:06.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:06.704048 17322 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu
16:12:06.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu
16:12:06.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-0/wal
16:12:06.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-0/data
16:12:06.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
16:12:06.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
16:12:06.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
16:12:06.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:12:06.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:12:06.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:12:06.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-0/logs
16:12:06.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-0/data/info.pb
16:12:06.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:12:06.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
16:12:06.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
16:12:06.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
16:12:06.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
16:12:06.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:12:06.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:12:06.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:12:06.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
16:12:06.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
16:12:06.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.234.129:42901
16:12:06.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.16.234.129
16:12:06.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.16.234.190:33901,127.16.234.189:32987,127.16.234.188:38065
16:12:06.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=43191
16:12:06.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.234.129
16:12:06.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.234.148:43519
16:12:06.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:12:06.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:12:06.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
16:12:07.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:12:07.014732 21524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
16:12:07.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:12:07.015452 21524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
16:12:07.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:12:07.016184 21524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
16:12:07.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:12:07.060953 21524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
16:12:07.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:12:07.061795 21524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.234.129
16:12:07.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:07.098341 21524 tablet_server_runner.cc:78] Tablet server non-default flags:
16:12:07.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:12:07.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.234.148:43519
16:12:07.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:12:07.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
16:12:07.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-0/data
16:12:07.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-0/wal
16:12:07.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:12:07.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:12:07.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.234.129:42901
16:12:07.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
16:12:07.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:12:07.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:12:07.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-0/data/info.pb
16:12:07.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.234.129
16:12:07.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=43191
16:12:07.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.16.234.190:33901,127.16.234.189:32987,127.16.234.188:38065
16:12:07.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
16:12:07.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:12:07.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
16:12:07.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
16:12:07.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:12:07.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.16.234.129
16:12:07.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-0/logs
16:12:07.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:12:07.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
16:12:07.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
16:12:07.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
16:12:07.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
16:12:07.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
16:12:07.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
16:12:07.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 15:12:52 UTC on ccycloud.quasar-tzrfpd.root.comops.site
16:12:07.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4580143
16:12:07.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
16:12:07.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:07.099527 21524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
16:12:07.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:07.101099 21524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
16:12:07.120 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:12:07.119571 21540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:12:08.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:12:07.119771 21550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:12:08.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:12:07.120064 21542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:12:08.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:12:08.232635 21546 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
16:12:08.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:08.232754 21524 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
16:12:08.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:08.236572 21524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
16:12:08.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:08.239243 21524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
16:12:08.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:08.240694 21524 hybrid_clock.cc:648] HybridClock initialized: now 1786464728240634 us; error 68 us; skew 500 ppm
16:12:08.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:08.241564 21524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
16:12:08.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:08.248047 21524 webserver.cc:460] Webserver started at http://127.16.234.129:43191/ using document root <none> and password file <none>
16:12:08.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:08.248939 21524 fs_manager.cc:359] Metadata directory not provided
16:12:08.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:08.249141 21524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
16:12:08.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:08.257239 21524 fs_manager.cc:709] Time spent opening directory manager: real 0.005s	user 0.005s	sys 0.000s
16:12:08.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:08.262142 21569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
16:12:08.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:08.263564 21524 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
16:12:08.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:08.263841 21524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-0/data,/tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-0/wal
16:12:08.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "d9315253ea9e464c9416773b52ee00a6"
16:12:08.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 16:11:14 on dist-test-slave-zj8q"
16:12:08.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:08.266233 21524 fs_report.cc:383] FS layout report
16:12:08.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
16:12:08.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-0/wal
16:12:08.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-0/wal
16:12:08.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-0/data/data
16:12:08.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
16:12:08.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
16:12:08.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
16:12:08.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
16:12:08.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
16:12:08.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
16:12:08.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
16:12:08.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
16:12:08.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
16:12:08.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
16:12:08.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
16:12:08.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:08.304641 21524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
16:12:08.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:08.306115 21524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
16:12:08.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:08.306586 21524 kserver.cc:163] Server-wide thread pool size limit: 3276
16:12:08.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:08.309089 21524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
16:12:08.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:08.315511 21580 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
16:12:08.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:08.323462 21524 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
16:12:08.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:08.323689 21524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s	user 0.000s	sys 0.002s
16:12:08.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:08.323900 21524 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
16:12:08.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:08.328668 21524 ts_tablet_manager.cc:610] Registered 1 tablets
16:12:08.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:08.328842 21524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s	user 0.002s	sys 0.001s
16:12:08.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:08.329249 21580 tablet_bootstrap.cc:492] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6: Bootstrap starting.
16:12:08.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:08.407713 21580 log.cc:824] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6: Log is configured to *not* fsync() on all Append() calls
16:12:08.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:08.543787 21580 tablet_bootstrap.cc:492] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6: Bootstrap replayed 1/1 log segments. Stats: ops{read=40 overwritten=0 applied=40 ignored=0} inserts{seen=30 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
16:12:08.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:08.544842 21580 tablet_bootstrap.cc:492] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6: Bootstrap complete.
16:12:08.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:08.546687 21580 ts_tablet_manager.cc:1397] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6: Time spent bootstrapping tablet: real 0.218s	user 0.146s	sys 0.064s
16:12:08.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:08.560652 21524 rpc_server.cc:307] RPC server started. Bound to: 127.16.234.129:42901
16:12:08.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:08.561499 21761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.234.129:42901 every 8 connection(s)
16:12:08.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:08.564749 21524 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklXYFrv/test-tmp/mini-kudu-cluster4809752246121975442/ts-0/data/info.pb
16:12:08.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:08.569000 17322 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu as pid 21524
16:12:08.570 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
16:12:08.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:08.582221 21580 raft_consensus.cc:348] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 11 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:12:08.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:08.587481 21580 raft_consensus.cc:729] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [term 11 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9315253ea9e464c9416773b52ee00a6, State: Initialized, Role: FOLLOWER
16:12:08.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:08.588864 21580 consensus_queue.cc:260] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 40, Last appended: 11.40, Last appended by leader: 40, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d392ba816c24499a0f7fecd95f88c15" member_type: VOTER last_known_addr { host: "127.16.234.131" port: 42843 } } peers { permanent_uuid: "d9315253ea9e464c9416773b52ee00a6" member_type: VOTER last_known_addr { host: "127.16.234.129" port: 42901 } } peers { permanent_uuid: "1bce670edc6c40b2bb1584ec3b364e0f" member_type: VOTER last_known_addr { host: "127.16.234.130" port: 43937 } }
16:12:08.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:08.603276 21580 ts_tablet_manager.cc:1428] T 7d3c128d101444a59e009013a5c634dc P d9315253ea9e464c9416773b52ee00a6: Time spent starting tablet: real 0.056s	user 0.027s	sys 0.021s
16:12:08.605 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 33 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304, tablet=7d3c128d101444a59e009013a5c634dc, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
16:12:08.630 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 33 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786464679304, tablet=7d3c128d101444a59e009013a5c634dc, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
16:12:08.641 [DEBUG - main] (Connection.java:575) [peer 1bce670edc6c40b2bb1584ec3b364e0f(127.16.234.130:43937)] disconnecting while in state TERMINATED
16:12:08.643 [DEBUG - main] (Connection.java:575) [peer d9315253ea9e464c9416773b52ee00a6(127.16.234.129:42901)] disconnecting while in state TERMINATED
16:12:08.644 [DEBUG - main] (Connection.java:575) [peer master-127.16.234.189:32987(127.16.234.189:32987)] disconnecting while in state READY
16:12:08.654 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.16.234.189:32987(127.16.234.189:32987)] handling channelInactive
16:12:08.655 [DEBUG - main] (Connection.java:575) [peer master-127.16.234.190:33901(127.16.234.190:33901)] disconnecting while in state READY
16:12:08.655 [DEBUG - main] (Connection.java:575) [peer master-127.16.234.188:38065(127.16.234.188:38065)] disconnecting while in state READY
16:12:08.655 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.16.234.189:32987(127.16.234.189:32987)] cleaning up while in state READY due to: connection closed
16:12:08.657 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.16.234.190:33901(127.16.234.190:33901)] handling channelInactive
16:12:08.657 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.16.234.190:33901(127.16.234.190:33901)] cleaning up while in state READY due to: connection closed
16:12:08.658 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.16.234.188:38065(127.16.234.188:38065)] handling channelInactive
16:12:08.658 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.16.234.188:38065(127.16.234.188:38065)] cleaning up while in state READY due to: connection closed
16:12:08.659 [DEBUG - main] (Connection.java:575) [peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)] disconnecting while in state READY
16:12:08.663 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2704) Releasing all remaining resources
16:12:08.670 [DEBUG - kudu-nio-0] (Connection.java:239) [peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)] handling channelInactive
16:12:08.670 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 3d392ba816c24499a0f7fecd95f88c15(127.16.234.131:42843)] cleaning up while in state READY due to: connection closed
16:12:08.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:08.671741 17322 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu with pid 21524
16:12:08.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:08.701735 17322 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu with pid 21242
16:12:08.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:12:08.703873 21776 negotiation.cc:337] Failed RPC negotiation. Trace:
16:12:08.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:12:08.634283 (+     0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.234.129:56637 (local address 127.16.234.190:33901)
16:12:08.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:12:08.677947 (+ 43664us) server_negotiation.cc:206] Beginning negotiation
16:12:08.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:12:08.678007 (+    60us) server_negotiation.cc:399] Waiting for connection header
16:12:08.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:12:08.701781 (+ 23774us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.234.129:56637: BlockingRecv error: recv got EOF from 127.16.234.129:56637 (error 108)
16:12:08.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":42571,"thread_start_us":47755,"threads_started":1}
16:12:08.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:12:08.705976 21766 negotiation.cc:337] Failed RPC negotiation. Trace:
16:12:08.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:12:08.583258 (+     0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.234.129:51973 (local address 127.16.234.188:38065)
16:12:08.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:12:08.595836 (+ 12578us) server_negotiation.cc:206] Beginning negotiation
16:12:08.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:12:08.595913 (+    77us) server_negotiation.cc:399] Waiting for connection header
16:12:08.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:12:08.606194 (+ 10281us) server_negotiation.cc:407] Connection header received
16:12:08.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:12:08.606478 (+   284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
16:12:08.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:12:08.606518 (+    40us) server_negotiation.cc:461] Received NEGOTIATE request from client
16:12:08.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:12:08.607051 (+   533us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
16:12:08.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:12:08.607384 (+   333us) server_negotiation.cc:226] Negotiated authn=SASL
16:12:08.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:12:08.652221 (+ 44837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
16:12:08.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:12:08.654854 (+  2633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
16:12:08.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:12:08.704839 (+ 49985us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.234.129:51973: BlockingRecv error: recv got EOF from 127.16.234.129:51973 (error 108)
16:12:08.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":11747,"thread_start_us":11278,"threads_started":1}
16:12:08.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:12:08.708901 21772 negotiation.cc:337] Failed RPC negotiation. Trace:
16:12:08.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:12:08.621127 (+     0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.234.129:46327 (local address 127.16.234.189:32987)
16:12:08.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:12:08.665369 (+ 44242us) server_negotiation.cc:206] Beginning negotiation
16:12:08.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:12:08.665431 (+    62us) server_negotiation.cc:399] Waiting for connection header
16:12:08.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:12:08.665511 (+    80us) server_negotiation.cc:407] Connection header received
16:12:08.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:12:08.665805 (+   294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
16:12:08.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:12:08.665863 (+    58us) server_negotiation.cc:461] Received NEGOTIATE request from client
16:12:08.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:12:08.666354 (+   491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
16:12:08.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:12:08.666695 (+   341us) server_negotiation.cc:226] Negotiated authn=SASL
16:12:08.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:12:08.672749 (+  6054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
16:12:08.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:12:08.675271 (+  2522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
16:12:08.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:12:08.707109 (+ 31838us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.234.129:46327: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
16:12:08.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":43328,"thread_start_us":51620,"threads_started":1}
16:12:08.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:08.739248 17322 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu with pid 20929
16:12:08.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:08.776206 17322 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu with pid 17347
16:12:08.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:08.813959 17322 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu with pid 17473
16:12:08.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:12:08.831679 17645 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.234.190:33901: connect: Connection refused (error 111)
16:12:08.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:12:08.834753 17645 consensus_peers.cc:487] T 00000000000000000000000000000000 P cdd30a1bb4e2432e9b560c791116b727 -> Peer 2ea4ce510c534787afa58a33455b3bff (127.16.234.190:33901): Couldn't send request to peer 2ea4ce510c534787afa58a33455b3bff. Status: Network error: Client connection negotiation failed: client connection to 127.16.234.190:33901: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
16:12:08.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:12:08.853300 17322 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklXYFrv/build/tsan/bin/kudu with pid 17596
16:12:09.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-08-11T16:12:09Z chronyd exiting

Time: 65.466

OK (1 test)

