JUnit version 4.13.2
.00:14:22.145 [DEBUG - main] (RandomUtils.java:49) Using random seed: 1786493662137
00:14:22.208 [INFO - main] (RetryRule.java:138) Creating RetryStatement with result reporter and retry count of 0 (no retries)
00:14:22.244 [INFO - main] (KuduTestHarness.java:155) Creating a new MiniKuduCluster...
00:14:22.258 [INFO - main] (TempDirUtils.java:71) Using the temp directory defined by TEST_TMPDIR: /tmp/dist-test-taskq35Ux4/test-tmp
00:14:22.287 [INFO - main] (KuduBinaryLocator.java:83) Using Kudu binary directory specified by system property 'kuduBinDir': ../../../build/tsan/bin
00:14:22.291 [INFO - main] (MiniKuduCluster.java:217) Starting process: [/tmp/dist-test-taskq35Ux4/java/build/dist-test/../../../build/tsan/bin/kudu, test, mini_cluster, --serialization=pb]
00:14:22.529 [DEBUG - main] (MiniKuduCluster.java:185) Request: create_cluster {
  num_masters: 3
  num_tservers: 3
  enable_kerberos: false
  cluster_root: "/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778"
  hms_mode: NONE
  mini_kdc_options {
  }
  principal: "kudu"
  mini_oidc_options {
  }
}

00:14:22.838 [DEBUG - main] (MiniKuduCluster.java:195) Response: 
00:14:22.839 [DEBUG - main] (MiniKuduCluster.java:185) Request: start_cluster {
}

00:14:22.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:22.846899 29300 mini_oidc.cc:150] Starting JWKS server
00:14:22.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:22.853902 29300 webserver.cc:533] Webserver started at http://127.0.0.1:35141/ using document root <none> and password file <none>
00:14:22.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:22.854581 29300 mini_oidc.cc:186] Starting OIDC Discovery server
00:14:22.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:22.857013 29300 webserver.cc:533] Webserver started at http://127.0.0.1:40529/ using document root <none> and password file <none>
00:14:22.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-08-12T00:14:22Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
00:14:22.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-08-12T00:14:22Z Disabled control of system clock
00:14:22.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:22.933610 29300 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu
00:14:22.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu
00:14:22.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/master-0/wal
00:14:22.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/master-0/data
00:14:22.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
00:14:22.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
00:14:22.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
00:14:22.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
00:14:22.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
00:14:22.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
00:14:22.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/master-0/logs
00:14:22.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/master-0/data/info.pb
00:14:22.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
00:14:22.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
00:14:22.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
00:14:22.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
00:14:22.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
00:14:22.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
00:14:22.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
00:14:22.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
00:14:22.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
00:14:22.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
00:14:22.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
00:14:22.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
00:14:22.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.28.157.62:35965
00:14:22.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.28.157.62
00:14:22.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
00:14:22.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.28.157.20:34671
00:14:22.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
00:14:22.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
00:14:22.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
00:14:22.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
00:14:22.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.28.157.62:35965,127.28.157.61:44801,127.28.157.60:33393
00:14:22.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
00:14:22.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:35141/jwks.json with env {}
00:14:23.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:23.266793 29314 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
00:14:23.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:23.267516 29314 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
00:14:23.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:23.268324 29314 flags.cc:432] Enabled unsafe flag: --never_fsync=true
00:14:23.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:23.303699 29314 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
00:14:23.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:23.304083 29314 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
00:14:23.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:23.304402 29314 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
00:14:23.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:23.304749 29314 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
00:14:23.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:23.305117 29314 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
00:14:23.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:23.305389 29314 flags.cc:432] Enabled experimental flag: --jwks_url=http://localhost:35141/jwks.json
00:14:23.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:23.344760 29314 master_runner.cc:387] Master server non-default flags:
00:14:23.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/master-0/logs
00:14:23.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
00:14:23.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
00:14:23.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
00:14:23.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.28.157.20:34671
00:14:23.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
00:14:23.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
00:14:23.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/master-0/data
00:14:23.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/master-0/wal
00:14:23.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
00:14:23.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.28.157.62:35965,127.28.157.61:44801,127.28.157.60:33393
00:14:23.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
00:14:23.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
00:14:23.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
00:14:23.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.28.157.62:35965
00:14:23.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
00:14:23.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
00:14:23.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
00:14:23.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:35141/jwks.json
00:14:23.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
00:14:23.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
00:14:23.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/master-0/data/info.pb
00:14:23.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.28.157.62
00:14:23.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
00:14:23.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
00:14:23.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
00:14:23.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
00:14:23.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
00:14:23.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
00:14:23.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
00:14:23.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
00:14:23.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
00:14:23.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision f7ed5b56c096f54e2f926bf94ceb7fc44c0c1329
00:14:23.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
00:14:23.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 11 Aug 2026 23:55:14 UTC on e7f111948823
00:14:23.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 17639
00:14:23.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
00:14:23.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:23.347855 29314 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
00:14:23.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:23.351032 29314 file_cache.cc:504] Constructed file cache file cache with capacity 419430
00:14:23.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:23.369527 29320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
00:14:24.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:23.369518 29322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
00:14:24.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:23.372845 29325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
00:14:24.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:24.632375 29324 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1263 milliseconds
00:14:24.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:24.632640 29314 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
00:14:24.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:24.637542 29314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
00:14:24.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:24.641338 29314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
00:14:24.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:24.643066 29314 hybrid_clock.cc:648] HybridClock initialized: now 1786493664643029 us; error 150 us; skew 500 ppm
00:14:24.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:24.644392 29314 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
00:14:24.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:24.652155 29314 webserver.cc:533] Webserver started at http://127.28.157.62:37877/ using document root <none> and password file <none>
00:14:24.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:24.654042 29314 fs_manager.cc:362] Metadata directory not provided
00:14:24.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:24.654306 29314 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
00:14:24.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:24.654949 29314 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
00:14:24.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:24.662228 29314 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/master-0/data/instance:
00:14:24.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "22c0f0f5cbfc4ad683b142e008a47923"
00:14:24.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 00:14:24 on dist-test-slave-t7g5"
00:14:24.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:24.664481 29314 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/master-0/wal/instance:
00:14:24.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "22c0f0f5cbfc4ad683b142e008a47923"
00:14:24.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 00:14:24 on dist-test-slave-t7g5"
00:14:24.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:24.675776 29314 fs_manager.cc:696] Time spent creating directory manager: real 0.010s	user 0.007s	sys 0.001s
00:14:24.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:24.686779 29336 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
00:14:24.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:24.689045 29314 fs_manager.cc:730] Time spent opening block manager: real 0.008s	user 0.007s	sys 0.001s
00:14:24.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:24.689512 29314 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/master-0/data,/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/master-0/wal
00:14:24.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "22c0f0f5cbfc4ad683b142e008a47923"
00:14:24.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 00:14:24 on dist-test-slave-t7g5"
00:14:24.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:24.689946 29314 fs_report.cc:389] FS layout report
00:14:24.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
00:14:24.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/master-0/wal
00:14:24.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/master-0/wal
00:14:24.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/master-0/data/data
00:14:24.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
00:14:24.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
00:14:24.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
00:14:24.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
00:14:24.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
00:14:24.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
00:14:24.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
00:14:24.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
00:14:24.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
00:14:24.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
00:14:24.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
00:14:24.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:24.744045 29314 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
00:14:24.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:24.745996 29314 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
00:14:24.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:24.746492 29314 kserver.cc:163] Server-wide thread pool size limit: 3276
00:14:24.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:24.835616 29314 rpc_server.cc:307] RPC server started. Bound to: 127.28.157.62:35965
00:14:24.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:24.835700 29414 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.157.62:35965 every 8 connection(s)
00:14:24.839 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:24.839300 29314 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/master-0/data/info.pb
00:14:24.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:24.841387 29300 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu as pid 29314
00:14:24.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:24.842056 29300 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/master-0/wal/instance
00:14:24.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:24.847024 29300 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu
00:14:24.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu
00:14:24.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/master-1/wal
00:14:24.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/master-1/data
00:14:24.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
00:14:24.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
00:14:24.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
00:14:24.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
00:14:24.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
00:14:24.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
00:14:24.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/master-1/logs
00:14:24.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/master-1/data/info.pb
00:14:24.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
00:14:24.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
00:14:24.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
00:14:24.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
00:14:24.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
00:14:24.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
00:14:24.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
00:14:24.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
00:14:24.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
00:14:24.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
00:14:24.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
00:14:24.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
00:14:24.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.28.157.61:44801
00:14:24.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.28.157.61
00:14:24.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
00:14:24.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.28.157.20:34671
00:14:24.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
00:14:24.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
00:14:24.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
00:14:24.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
00:14:24.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.28.157.62:35965,127.28.157.61:44801,127.28.157.60:33393
00:14:24.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
00:14:24.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:35141/jwks.json with env {}
00:14:24.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:24.851686 29415 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
00:14:24.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:24.868438 29415 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.157.62" port: 35965 } has no permanent_uuid. Determining permanent_uuid...
00:14:24.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:24.909370 29415 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.157.61" port: 44801 } has no permanent_uuid. Determining permanent_uuid...
00:14:24.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:24.912343 29339 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.157.61:44801: connect: Connection refused (error 111)
00:14:24.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:24.917379 29415 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.28.157.61:44801: Network error: Client connection negotiation failed: client connection to 127.28.157.61:44801: connect: Connection refused (error 111)
00:14:24.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:24.966989 29415 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.157.61" port: 44801 } attempt: 1
00:14:24.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:24.971714 29415 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.28.157.61:44801: Network error: Client connection negotiation failed: client connection to 127.28.157.61:44801: connect: Connection refused (error 111)
00:14:25.041 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:25.040218 29415 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.157.61" port: 44801 } attempt: 2
00:14:25.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:25.045018 29415 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.28.157.61:44801: Network error: Client connection negotiation failed: client connection to 127.28.157.61:44801: connect: Connection refused (error 111)
00:14:25.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:25.136652 29415 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.157.61" port: 44801 } attempt: 3
00:14:25.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:25.141322 29415 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.28.157.61:44801: Network error: Client connection negotiation failed: client connection to 127.28.157.61:44801: connect: Connection refused (error 111)
00:14:25.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:25.203961 29419 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
00:14:25.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:25.204548 29419 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
00:14:25.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:25.205034 29419 flags.cc:432] Enabled unsafe flag: --never_fsync=true
00:14:25.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:25.239413 29419 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
00:14:25.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:25.239730 29419 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
00:14:25.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:25.239934 29419 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
00:14:25.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:25.240127 29419 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
00:14:25.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:25.240309 29419 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
00:14:25.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:25.240512 29419 flags.cc:432] Enabled experimental flag: --jwks_url=http://localhost:35141/jwks.json
00:14:25.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:25.284157 29419 master_runner.cc:387] Master server non-default flags:
00:14:25.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/master-1/logs
00:14:25.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
00:14:25.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
00:14:25.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
00:14:25.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.28.157.20:34671
00:14:25.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
00:14:25.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
00:14:25.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/master-1/data
00:14:25.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/master-1/wal
00:14:25.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
00:14:25.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.28.157.62:35965,127.28.157.61:44801,127.28.157.60:33393
00:14:25.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
00:14:25.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
00:14:25.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
00:14:25.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.28.157.61:44801
00:14:25.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
00:14:25.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
00:14:25.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
00:14:25.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:35141/jwks.json
00:14:25.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
00:14:25.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
00:14:25.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/master-1/data/info.pb
00:14:25.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.28.157.61
00:14:25.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
00:14:25.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
00:14:25.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
00:14:25.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
00:14:25.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
00:14:25.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
00:14:25.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
00:14:25.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
00:14:25.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
00:14:25.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision f7ed5b56c096f54e2f926bf94ceb7fc44c0c1329
00:14:25.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
00:14:25.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 11 Aug 2026 23:55:14 UTC on e7f111948823
00:14:25.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 17639
00:14:25.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
00:14:25.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:25.284824 29415 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.157.61" port: 44801 } attempt: 4
00:14:25.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:25.285884 29419 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
00:14:25.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:25.289104 29419 file_cache.cc:504] Constructed file cache file cache with capacity 419430
00:14:25.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:25.292025 29415 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.28.157.61:44801: Network error: Client connection negotiation failed: client connection to 127.28.157.61:44801: connect: Connection refused (error 111)
00:14:25.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:25.302932 29434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
00:14:25.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:25.591521 29415 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.157.61" port: 44801 } attempt: 5
00:14:25.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:25.597936 29415 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.28.157.61:44801: Network error: Client connection negotiation failed: client connection to 127.28.157.61:44801: connect: Connection refused (error 111)
00:14:26.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:26.145462 29415 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.157.61" port: 44801 } attempt: 6
00:14:26.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:26.157546 29415 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.28.157.61:44801: Network error: Client connection negotiation failed: client connection to 127.28.157.61:44801: connect: Connection refused (error 111)
00:14:26.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:26.494071 29436 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
00:14:26.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:26.494939 29437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
00:14:26.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:25.303776 29435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
00:14:26.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:26.498832 29419 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.196s	user 0.442s	sys 0.753s
00:14:26.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:26.499123 29419 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.196s	user 0.442s	sys 0.753s
00:14:26.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:26.499415 29419 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
00:14:26.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:26.505254 29419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
00:14:26.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:26.510490 29419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
00:14:26.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:26.512584 29419 hybrid_clock.cc:648] HybridClock initialized: now 1786493666512497 us; error 93 us; skew 500 ppm
00:14:26.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:26.513631 29419 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
00:14:26.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:26.530617 29419 webserver.cc:533] Webserver started at http://127.28.157.61:41039/ using document root <none> and password file <none>
00:14:26.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:26.532567 29419 fs_manager.cc:362] Metadata directory not provided
00:14:26.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:26.533324 29419 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
00:14:26.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:26.534998 29419 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
00:14:26.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:26.540213 29419 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/master-1/data/instance:
00:14:26.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "011d9300ff864ff9b52a082026a4136a"
00:14:26.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 00:14:26 on dist-test-slave-t7g5"
00:14:26.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:26.542568 29419 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/master-1/wal/instance:
00:14:26.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "011d9300ff864ff9b52a082026a4136a"
00:14:26.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 00:14:26 on dist-test-slave-t7g5"
00:14:26.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:26.552057 29419 fs_manager.cc:696] Time spent creating directory manager: real 0.008s	user 0.003s	sys 0.003s
00:14:26.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:26.559343 29455 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
00:14:26.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:26.560899 29419 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.001s	sys 0.004s
00:14:26.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:26.561275 29419 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/master-1/data,/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/master-1/wal
00:14:26.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "011d9300ff864ff9b52a082026a4136a"
00:14:26.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 00:14:26 on dist-test-slave-t7g5"
00:14:26.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:26.561815 29419 fs_report.cc:389] FS layout report
00:14:26.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
00:14:26.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/master-1/wal
00:14:26.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/master-1/wal
00:14:26.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/master-1/data/data
00:14:26.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
00:14:26.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
00:14:26.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
00:14:26.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
00:14:26.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
00:14:26.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
00:14:26.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
00:14:26.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
00:14:26.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
00:14:26.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
00:14:26.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
00:14:26.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:26.638990 29419 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
00:14:26.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:26.640578 29419 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
00:14:26.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:26.641121 29419 kserver.cc:163] Server-wide thread pool size limit: 3276
00:14:26.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:26.720077 29419 rpc_server.cc:307] RPC server started. Bound to: 127.28.157.61:44801
00:14:26.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:26.720184 29536 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.157.61:44801 every 8 connection(s)
00:14:26.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:26.723922 29419 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/master-1/data/info.pb
00:14:26.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:26.725517 29300 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu as pid 29419
00:14:26.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:26.726293 29300 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/master-1/wal/instance
00:14:26.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:26.729513 29300 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu
00:14:26.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu
00:14:26.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/master-2/wal
00:14:26.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/master-2/data
00:14:26.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
00:14:26.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
00:14:26.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
00:14:26.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
00:14:26.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
00:14:26.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
00:14:26.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/master-2/logs
00:14:26.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/master-2/data/info.pb
00:14:26.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
00:14:26.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
00:14:26.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
00:14:26.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
00:14:26.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
00:14:26.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
00:14:26.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
00:14:26.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
00:14:26.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
00:14:26.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
00:14:26.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
00:14:26.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
00:14:26.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.28.157.60:33393
00:14:26.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.28.157.60
00:14:26.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
00:14:26.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.28.157.20:34671
00:14:26.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
00:14:26.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
00:14:26.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
00:14:26.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
00:14:26.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.28.157.62:35965,127.28.157.61:44801,127.28.157.60:33393
00:14:26.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
00:14:26.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:35141/jwks.json with env {}
00:14:26.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:26.741109 29538 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
00:14:26.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:26.753799 29538 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.157.62" port: 35965 } has no permanent_uuid. Determining permanent_uuid...
00:14:26.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:26.786401 29538 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.157.61" port: 44801 } has no permanent_uuid. Determining permanent_uuid...
00:14:26.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:26.802974 29538 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.157.60" port: 33393 } has no permanent_uuid. Determining permanent_uuid...
00:14:26.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:26.805944 29456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.157.60:33393: connect: Connection refused (error 111)
00:14:26.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:26.811167 29538 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.28.157.60:33393: Network error: Client connection negotiation failed: client connection to 127.28.157.60:33393: connect: Connection refused (error 111)
00:14:26.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:26.860903 29538 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.157.60" port: 33393 } attempt: 1
00:14:26.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:26.865372 29538 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.28.157.60:33393: Network error: Client connection negotiation failed: client connection to 127.28.157.60:33393: connect: Connection refused (error 111)
00:14:26.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:26.933928 29538 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.157.60" port: 33393 } attempt: 2
00:14:26.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:26.939189 29538 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.28.157.60:33393: Network error: Client connection negotiation failed: client connection to 127.28.157.60:33393: connect: Connection refused (error 111)
00:14:27.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:27.030764 29538 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.157.60" port: 33393 } attempt: 3
00:14:27.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:27.035760 29538 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.28.157.60:33393: Network error: Client connection negotiation failed: client connection to 127.28.157.60:33393: connect: Connection refused (error 111)
00:14:27.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:27.083112 29542 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
00:14:27.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:27.083702 29542 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
00:14:27.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:27.084246 29542 flags.cc:432] Enabled unsafe flag: --never_fsync=true
00:14:27.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:27.118835 29542 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
00:14:27.120 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:27.119163 29542 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
00:14:27.120 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:27.119371 29542 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
00:14:27.120 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:27.119570 29542 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
00:14:27.120 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:27.119756 29542 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
00:14:27.120 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:27.119971 29542 flags.cc:432] Enabled experimental flag: --jwks_url=http://localhost:35141/jwks.json
00:14:27.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:27.161092 29542 master_runner.cc:387] Master server non-default flags:
00:14:27.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/master-2/logs
00:14:27.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
00:14:27.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
00:14:27.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
00:14:27.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.28.157.20:34671
00:14:27.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
00:14:27.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
00:14:27.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/master-2/data
00:14:27.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/master-2/wal
00:14:27.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
00:14:27.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.28.157.62:35965,127.28.157.61:44801,127.28.157.60:33393
00:14:27.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
00:14:27.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
00:14:27.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
00:14:27.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.28.157.60:33393
00:14:27.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
00:14:27.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
00:14:27.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
00:14:27.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:35141/jwks.json
00:14:27.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
00:14:27.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
00:14:27.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/master-2/data/info.pb
00:14:27.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.28.157.60
00:14:27.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
00:14:27.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
00:14:27.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
00:14:27.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
00:14:27.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
00:14:27.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
00:14:27.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
00:14:27.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
00:14:27.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
00:14:27.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision f7ed5b56c096f54e2f926bf94ceb7fc44c0c1329
00:14:27.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
00:14:27.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 11 Aug 2026 23:55:14 UTC on e7f111948823
00:14:27.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 17639
00:14:27.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
00:14:27.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:27.162567 29542 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
00:14:27.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:27.164271 29542 file_cache.cc:504] Constructed file cache file cache with capacity 419430
00:14:27.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:27.176602 29559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
00:14:27.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:27.179299 29538 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.157.60" port: 33393 } attempt: 4
00:14:27.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:27.186354 29538 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.28.157.60:33393: Network error: Client connection negotiation failed: client connection to 127.28.157.60:33393: connect: Connection refused (error 111)
00:14:27.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:27.218104 29415 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.157.61" port: 44801 } attempt: 7
00:14:27.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:27.250768 29415 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.157.60" port: 33393 } has no permanent_uuid. Determining permanent_uuid...
00:14:27.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:27.279119 29415 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.28.157.60:33393: Network error: Client connection negotiation failed: client connection to 127.28.157.60:33393: connect: Connection refused (error 111)
00:14:27.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:27.337807 29415 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.157.60" port: 33393 } attempt: 1
00:14:27.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:27.345829 29415 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.28.157.60:33393: Network error: Client connection negotiation failed: client connection to 127.28.157.60:33393: connect: Connection refused (error 111)
00:14:27.428 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:27.427392 29415 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.157.60" port: 33393 } attempt: 2
00:14:27.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:27.436273 29415 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.28.157.60:33393: Network error: Client connection negotiation failed: client connection to 127.28.157.60:33393: connect: Connection refused (error 111)
00:14:27.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:27.485881 29538 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.157.60" port: 33393 } attempt: 5
00:14:27.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:27.498307 29538 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.28.157.60:33393: Network error: Client connection negotiation failed: client connection to 127.28.157.60:33393: connect: Connection refused (error 111)
00:14:27.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:27.521849 29415 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.157.60" port: 33393 } attempt: 3
00:14:27.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:27.534186 29415 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.28.157.60:33393: Network error: Client connection negotiation failed: client connection to 127.28.157.60:33393: connect: Connection refused (error 111)
00:14:27.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:27.674746 29415 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.157.60" port: 33393 } attempt: 4
00:14:27.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:27.683354 29415 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.28.157.60:33393: Network error: Client connection negotiation failed: client connection to 127.28.157.60:33393: connect: Connection refused (error 111)
00:14:27.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:27.966897 29415 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.157.60" port: 33393 } attempt: 5
00:14:27.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:27.975018 29415 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.28.157.60:33393: Network error: Client connection negotiation failed: client connection to 127.28.157.60:33393: connect: Connection refused (error 111)
00:14:28.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:28.045979 29538 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.157.60" port: 33393 } attempt: 6
00:14:28.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:28.078287 29538 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.28.157.60:33393: Network error: Client connection negotiation failed: client connection to 127.28.157.60:33393: connect: Connection refused (error 111)
00:14:28.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:27.178421 29560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
00:14:28.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:28.416653 29562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
00:14:28.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:28.419050 29561 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1236 milliseconds
00:14:28.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:28.419899 29542 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.242s	user 0.462s	sys 0.775s
00:14:28.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:28.420252 29542 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.243s	user 0.462s	sys 0.775s
00:14:28.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:28.420548 29542 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
00:14:28.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:28.422130 29542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
00:14:28.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:28.425457 29542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
00:14:28.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:28.426968 29542 hybrid_clock.cc:648] HybridClock initialized: now 1786493668426889 us; error 107 us; skew 500 ppm
00:14:28.428 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:28.428112 29542 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
00:14:28.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:28.437680 29542 webserver.cc:533] Webserver started at http://127.28.157.60:37633/ using document root <none> and password file <none>
00:14:28.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:28.439123 29542 fs_manager.cc:362] Metadata directory not provided
00:14:28.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:28.439455 29542 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
00:14:28.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:28.440045 29542 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
00:14:28.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:28.446785 29542 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/master-2/data/instance:
00:14:28.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "b849676f6ae3499e801495a62dc3b393"
00:14:28.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 00:14:28 on dist-test-slave-t7g5"
00:14:28.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:28.448439 29542 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/master-2/wal/instance:
00:14:28.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "b849676f6ae3499e801495a62dc3b393"
00:14:28.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 00:14:28 on dist-test-slave-t7g5"
00:14:28.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:28.459234 29542 fs_manager.cc:696] Time spent creating directory manager: real 0.010s	user 0.010s	sys 0.002s
00:14:28.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:28.467411 29580 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
00:14:28.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:28.468930 29542 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.002s	sys 0.003s
00:14:28.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:28.469370 29542 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/master-2/data,/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/master-2/wal
00:14:28.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "b849676f6ae3499e801495a62dc3b393"
00:14:28.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 00:14:28 on dist-test-slave-t7g5"
00:14:28.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:28.469882 29542 fs_report.cc:389] FS layout report
00:14:28.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
00:14:28.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/master-2/wal
00:14:28.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/master-2/wal
00:14:28.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/master-2/data/data
00:14:28.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
00:14:28.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
00:14:28.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
00:14:28.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
00:14:28.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
00:14:28.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
00:14:28.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
00:14:28.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
00:14:28.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
00:14:28.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
00:14:28.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
00:14:28.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:28.527674 29415 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.157.60" port: 33393 } attempt: 6
00:14:28.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:28.535594 29415 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.28.157.60:33393: Network error: Client connection negotiation failed: client connection to 127.28.157.60:33393: connect: Connection refused (error 111)
00:14:28.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:28.563676 29542 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
00:14:28.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:28.565210 29542 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
00:14:28.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:28.565685 29542 kserver.cc:163] Server-wide thread pool size limit: 3276
00:14:28.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:28.646116 29542 rpc_server.cc:307] RPC server started. Bound to: 127.28.157.60:33393
00:14:28.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:28.646453 29674 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.157.60:33393 every 8 connection(s)
00:14:28.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:28.649111 29542 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/master-2/data/info.pb
00:14:28.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:28.655714 29676 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
00:14:28.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:28.658597 29300 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu as pid 29542
00:14:28.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:28.658988 29300 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/master-2/wal/instance
00:14:28.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:28.669972 29676 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.157.62" port: 35965 } has no permanent_uuid. Determining permanent_uuid...
00:14:28.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:28.701889 29676 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.157.61" port: 44801 } has no permanent_uuid. Determining permanent_uuid...
00:14:28.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:28.713189 29676 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.157.60" port: 33393 } has no permanent_uuid. Determining permanent_uuid...
00:14:28.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:28.737464 29676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b849676f6ae3499e801495a62dc3b393: Bootstrap starting.
00:14:28.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:28.748955 29676 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b849676f6ae3499e801495a62dc3b393: Neither blocks nor log segments found. Creating new log.
00:14:28.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:28.753139 29676 log.cc:826] T 00000000000000000000000000000000 P b849676f6ae3499e801495a62dc3b393: Log is configured to *not* fsync() on all Append() calls
00:14:28.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:28.759364 29676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b849676f6ae3499e801495a62dc3b393: No bootstrap required, opened a new log
00:14:28.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:28.782869 29676 raft_consensus.cc:359] T 00000000000000000000000000000000 P b849676f6ae3499e801495a62dc3b393 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22c0f0f5cbfc4ad683b142e008a47923" member_type: VOTER last_known_addr { host: "127.28.157.62" port: 35965 } } peers { permanent_uuid: "011d9300ff864ff9b52a082026a4136a" member_type: VOTER last_known_addr { host: "127.28.157.61" port: 44801 } } peers { permanent_uuid: "b849676f6ae3499e801495a62dc3b393" member_type: VOTER last_known_addr { host: "127.28.157.60" port: 33393 } }
00:14:28.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:28.783574 29676 raft_consensus.cc:385] T 00000000000000000000000000000000 P b849676f6ae3499e801495a62dc3b393 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
00:14:28.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:28.783871 29676 raft_consensus.cc:740] T 00000000000000000000000000000000 P b849676f6ae3499e801495a62dc3b393 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b849676f6ae3499e801495a62dc3b393, State: Initialized, Role: FOLLOWER
00:14:28.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:28.785351 29676 consensus_queue.cc:260] T 00000000000000000000000000000000 P b849676f6ae3499e801495a62dc3b393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22c0f0f5cbfc4ad683b142e008a47923" member_type: VOTER last_known_addr { host: "127.28.157.62" port: 35965 } } peers { permanent_uuid: "011d9300ff864ff9b52a082026a4136a" member_type: VOTER last_known_addr { host: "127.28.157.61" port: 44801 } } peers { permanent_uuid: "b849676f6ae3499e801495a62dc3b393" member_type: VOTER last_known_addr { host: "127.28.157.60" port: 33393 } }
00:14:28.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:28.789710 29688 sys_catalog.cc:455] T 00000000000000000000000000000000 P b849676f6ae3499e801495a62dc3b393 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22c0f0f5cbfc4ad683b142e008a47923" member_type: VOTER last_known_addr { host: "127.28.157.62" port: 35965 } } peers { permanent_uuid: "011d9300ff864ff9b52a082026a4136a" member_type: VOTER last_known_addr { host: "127.28.157.61" port: 44801 } } peers { permanent_uuid: "b849676f6ae3499e801495a62dc3b393" member_type: VOTER last_known_addr { host: "127.28.157.60" port: 33393 } } }
00:14:28.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:28.790800 29688 sys_catalog.cc:458] T 00000000000000000000000000000000 P b849676f6ae3499e801495a62dc3b393 [sys.catalog]: This master's current role is: FOLLOWER
00:14:28.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:28.792155 29676 sys_catalog.cc:565] T 00000000000000000000000000000000 P b849676f6ae3499e801495a62dc3b393 [sys.catalog]: configured and running, proceeding with master startup.
00:14:28.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:28.821444 29593 tablet.cc:2406] T 00000000000000000000000000000000 P b849676f6ae3499e801495a62dc3b393: Can't schedule compaction. Clean time has not been advanced past its initial value.
00:14:28.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:28.826870 29703 catalog_manager.cc:1594] T 00000000000000000000000000000000 P b849676f6ae3499e801495a62dc3b393: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
00:14:28.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:28.827288 29703 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
00:14:28.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:28.816728 29542 master_runner.cc:187] Error getting master registration for 127.28.157.62:35965: OK, instance_id { permanent_uuid: "22c0f0f5cbfc4ad683b142e008a47923" instance_seqno: 1786493664812820 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
00:14:28.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:28.845258 29542 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
00:14:29.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.045876 29688 raft_consensus.cc:493] T 00000000000000000000000000000000 P b849676f6ae3499e801495a62dc3b393 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
00:14:29.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.046574 29688 raft_consensus.cc:515] T 00000000000000000000000000000000 P b849676f6ae3499e801495a62dc3b393 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22c0f0f5cbfc4ad683b142e008a47923" member_type: VOTER last_known_addr { host: "127.28.157.62" port: 35965 } } peers { permanent_uuid: "011d9300ff864ff9b52a082026a4136a" member_type: VOTER last_known_addr { host: "127.28.157.61" port: 44801 } } peers { permanent_uuid: "b849676f6ae3499e801495a62dc3b393" member_type: VOTER last_known_addr { host: "127.28.157.60" port: 33393 } }
00:14:29.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.050257 29688 leader_election.cc:290] T 00000000000000000000000000000000 P b849676f6ae3499e801495a62dc3b393 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22c0f0f5cbfc4ad683b142e008a47923 (127.28.157.62:35965), 011d9300ff864ff9b52a082026a4136a (127.28.157.61:44801)
00:14:29.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.138839 29538 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.157.60" port: 33393 } attempt: 7
00:14:29.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.176349 29538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 011d9300ff864ff9b52a082026a4136a: Bootstrap starting.
00:14:29.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.185273 29538 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 011d9300ff864ff9b52a082026a4136a: Neither blocks nor log segments found. Creating new log.
00:14:29.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.187848 29538 log.cc:826] T 00000000000000000000000000000000 P 011d9300ff864ff9b52a082026a4136a: Log is configured to *not* fsync() on all Append() calls
00:14:29.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.193104 29538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 011d9300ff864ff9b52a082026a4136a: No bootstrap required, opened a new log
00:14:29.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.225425 29538 raft_consensus.cc:359] T 00000000000000000000000000000000 P 011d9300ff864ff9b52a082026a4136a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22c0f0f5cbfc4ad683b142e008a47923" member_type: VOTER last_known_addr { host: "127.28.157.62" port: 35965 } } peers { permanent_uuid: "011d9300ff864ff9b52a082026a4136a" member_type: VOTER last_known_addr { host: "127.28.157.61" port: 44801 } } peers { permanent_uuid: "b849676f6ae3499e801495a62dc3b393" member_type: VOTER last_known_addr { host: "127.28.157.60" port: 33393 } }
00:14:29.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.226735 29538 raft_consensus.cc:385] T 00000000000000000000000000000000 P 011d9300ff864ff9b52a082026a4136a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
00:14:29.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.228762 29538 raft_consensus.cc:740] T 00000000000000000000000000000000 P 011d9300ff864ff9b52a082026a4136a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 011d9300ff864ff9b52a082026a4136a, State: Initialized, Role: FOLLOWER
00:14:29.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.230662 29538 consensus_queue.cc:260] T 00000000000000000000000000000000 P 011d9300ff864ff9b52a082026a4136a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22c0f0f5cbfc4ad683b142e008a47923" member_type: VOTER last_known_addr { host: "127.28.157.62" port: 35965 } } peers { permanent_uuid: "011d9300ff864ff9b52a082026a4136a" member_type: VOTER last_known_addr { host: "127.28.157.61" port: 44801 } } peers { permanent_uuid: "b849676f6ae3499e801495a62dc3b393" member_type: VOTER last_known_addr { host: "127.28.157.60" port: 33393 } }
00:14:29.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.234493 29711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 011d9300ff864ff9b52a082026a4136a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22c0f0f5cbfc4ad683b142e008a47923" member_type: VOTER last_known_addr { host: "127.28.157.62" port: 35965 } } peers { permanent_uuid: "011d9300ff864ff9b52a082026a4136a" member_type: VOTER last_known_addr { host: "127.28.157.61" port: 44801 } } peers { permanent_uuid: "b849676f6ae3499e801495a62dc3b393" member_type: VOTER last_known_addr { host: "127.28.157.60" port: 33393 } } }
00:14:29.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.236056 29538 sys_catalog.cc:565] T 00000000000000000000000000000000 P 011d9300ff864ff9b52a082026a4136a [sys.catalog]: configured and running, proceeding with master startup.
00:14:29.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.236083 29711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 011d9300ff864ff9b52a082026a4136a [sys.catalog]: This master's current role is: FOLLOWER
00:14:29.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.236595 29500 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b849676f6ae3499e801495a62dc3b393" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "011d9300ff864ff9b52a082026a4136a" is_pre_election: true
00:14:29.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.238013 29500 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 011d9300ff864ff9b52a082026a4136a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b849676f6ae3499e801495a62dc3b393 in term 0.
00:14:29.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.240204 29590 leader_election.cc:304] T 00000000000000000000000000000000 P b849676f6ae3499e801495a62dc3b393 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 011d9300ff864ff9b52a082026a4136a, b849676f6ae3499e801495a62dc3b393; no voters: 
00:14:29.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.241390 29688 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b849676f6ae3499e801495a62dc3b393 [term 0 FOLLOWER]: Leader pre-election won for term 1
00:14:29.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.241784 29688 raft_consensus.cc:493] T 00000000000000000000000000000000 P b849676f6ae3499e801495a62dc3b393 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
00:14:29.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.242107 29688 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b849676f6ae3499e801495a62dc3b393 [term 0 FOLLOWER]: Advancing to term 1
00:14:29.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.253597 29688 raft_consensus.cc:515] T 00000000000000000000000000000000 P b849676f6ae3499e801495a62dc3b393 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22c0f0f5cbfc4ad683b142e008a47923" member_type: VOTER last_known_addr { host: "127.28.157.62" port: 35965 } } peers { permanent_uuid: "011d9300ff864ff9b52a082026a4136a" member_type: VOTER last_known_addr { host: "127.28.157.61" port: 44801 } } peers { permanent_uuid: "b849676f6ae3499e801495a62dc3b393" member_type: VOTER last_known_addr { host: "127.28.157.60" port: 33393 } }
00:14:29.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.258754 29500 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b849676f6ae3499e801495a62dc3b393" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "011d9300ff864ff9b52a082026a4136a"
00:14:29.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.261096 29500 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 011d9300ff864ff9b52a082026a4136a [term 0 FOLLOWER]: Advancing to term 1
00:14:29.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.277792 29688 leader_election.cc:290] T 00000000000000000000000000000000 P b849676f6ae3499e801495a62dc3b393 [CANDIDATE]: Term 1 election: Requested vote from peers 22c0f0f5cbfc4ad683b142e008a47923 (127.28.157.62:35965), 011d9300ff864ff9b52a082026a4136a (127.28.157.61:44801)
00:14:29.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.279768 29500 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 011d9300ff864ff9b52a082026a4136a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b849676f6ae3499e801495a62dc3b393 in term 1.
00:14:29.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.281148 29590 leader_election.cc:304] T 00000000000000000000000000000000 P b849676f6ae3499e801495a62dc3b393 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 011d9300ff864ff9b52a082026a4136a, b849676f6ae3499e801495a62dc3b393; no voters: 
00:14:29.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.282087 29688 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b849676f6ae3499e801495a62dc3b393 [term 1 FOLLOWER]: Leader election won for term 1
00:14:29.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.284204 29688 raft_consensus.cc:697] T 00000000000000000000000000000000 P b849676f6ae3499e801495a62dc3b393 [term 1 LEADER]: Becoming Leader. State: Replica: b849676f6ae3499e801495a62dc3b393, State: Running, Role: LEADER
00:14:29.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.285399 29688 consensus_queue.cc:237] T 00000000000000000000000000000000 P b849676f6ae3499e801495a62dc3b393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22c0f0f5cbfc4ad683b142e008a47923" member_type: VOTER last_known_addr { host: "127.28.157.62" port: 35965 } } peers { permanent_uuid: "011d9300ff864ff9b52a082026a4136a" member_type: VOTER last_known_addr { host: "127.28.157.61" port: 44801 } } peers { permanent_uuid: "b849676f6ae3499e801495a62dc3b393" member_type: VOTER last_known_addr { host: "127.28.157.60" port: 33393 } }
00:14:29.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.298947 29717 sys_catalog.cc:455] T 00000000000000000000000000000000 P b849676f6ae3499e801495a62dc3b393 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b849676f6ae3499e801495a62dc3b393. Latest consensus state: current_term: 1 leader_uuid: "b849676f6ae3499e801495a62dc3b393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22c0f0f5cbfc4ad683b142e008a47923" member_type: VOTER last_known_addr { host: "127.28.157.62" port: 35965 } } peers { permanent_uuid: "011d9300ff864ff9b52a082026a4136a" member_type: VOTER last_known_addr { host: "127.28.157.61" port: 44801 } } peers { permanent_uuid: "b849676f6ae3499e801495a62dc3b393" member_type: VOTER last_known_addr { host: "127.28.157.60" port: 33393 } } }
00:14:29.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.300398 29717 sys_catalog.cc:458] T 00000000000000000000000000000000 P b849676f6ae3499e801495a62dc3b393 [sys.catalog]: This master's current role is: LEADER
00:14:29.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.316184 29728 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
00:14:29.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.324287 29728 catalog_manager.cc:1520] Initializing Kudu cluster ID...
00:14:29.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.348658 29500 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 011d9300ff864ff9b52a082026a4136a [term 1 FOLLOWER]: Refusing update from remote peer b849676f6ae3499e801495a62dc3b393: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
00:14:29.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.352173 29717 consensus_queue.cc:1048] T 00000000000000000000000000000000 P b849676f6ae3499e801495a62dc3b393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "011d9300ff864ff9b52a082026a4136a" member_type: VOTER last_known_addr { host: "127.28.157.61" port: 44801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
00:14:29.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:29.385345 29738 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 011d9300ff864ff9b52a082026a4136a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
00:14:29.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:29.386943 29738 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
00:14:29.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:29.400024 29465 tablet.cc:2406] T 00000000000000000000000000000000 P 011d9300ff864ff9b52a082026a4136a: Can't schedule compaction. Clean time has not been advanced past its initial value.
00:14:29.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.355165 29419 master_runner.cc:187] Error getting master registration for 127.28.157.62:35965: OK, instance_id { permanent_uuid: "22c0f0f5cbfc4ad683b142e008a47923" instance_seqno: 1786493664812820 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
00:14:29.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.412436 29419 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
00:14:29.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.419945 29711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 011d9300ff864ff9b52a082026a4136a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b849676f6ae3499e801495a62dc3b393. Latest consensus state: current_term: 1 leader_uuid: "b849676f6ae3499e801495a62dc3b393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22c0f0f5cbfc4ad683b142e008a47923" member_type: VOTER last_known_addr { host: "127.28.157.62" port: 35965 } } peers { permanent_uuid: "011d9300ff864ff9b52a082026a4136a" member_type: VOTER last_known_addr { host: "127.28.157.61" port: 44801 } } peers { permanent_uuid: "b849676f6ae3499e801495a62dc3b393" member_type: VOTER last_known_addr { host: "127.28.157.60" port: 33393 } } }
00:14:29.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.426687 29711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 011d9300ff864ff9b52a082026a4136a [sys.catalog]: This master's current role is: FOLLOWER
00:14:29.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.431102 29750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 011d9300ff864ff9b52a082026a4136a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b849676f6ae3499e801495a62dc3b393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22c0f0f5cbfc4ad683b142e008a47923" member_type: VOTER last_known_addr { host: "127.28.157.62" port: 35965 } } peers { permanent_uuid: "011d9300ff864ff9b52a082026a4136a" member_type: VOTER last_known_addr { host: "127.28.157.61" port: 44801 } } peers { permanent_uuid: "b849676f6ae3499e801495a62dc3b393" member_type: VOTER last_known_addr { host: "127.28.157.60" port: 33393 } } }
00:14:29.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.431973 29750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 011d9300ff864ff9b52a082026a4136a [sys.catalog]: This master's current role is: FOLLOWER
00:14:29.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.442932 29717 sys_catalog.cc:455] T 00000000000000000000000000000000 P b849676f6ae3499e801495a62dc3b393 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b849676f6ae3499e801495a62dc3b393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22c0f0f5cbfc4ad683b142e008a47923" member_type: VOTER last_known_addr { host: "127.28.157.62" port: 35965 } } peers { permanent_uuid: "011d9300ff864ff9b52a082026a4136a" member_type: VOTER last_known_addr { host: "127.28.157.61" port: 44801 } } peers { permanent_uuid: "b849676f6ae3499e801495a62dc3b393" member_type: VOTER last_known_addr { host: "127.28.157.60" port: 33393 } } }
00:14:29.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.443696 29717 sys_catalog.cc:458] T 00000000000000000000000000000000 P b849676f6ae3499e801495a62dc3b393 [sys.catalog]: This master's current role is: LEADER
00:14:29.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.446084 29728 catalog_manager.cc:1383] Generated new cluster ID: d9ab6a5c52014b6a9f2ed508c8bca5a3
00:14:29.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.446403 29728 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
00:14:29.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.484047 29728 catalog_manager.cc:1406] Generated new certificate authority record
00:14:29.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.486796 29728 catalog_manager.cc:1540] Loading token signing keys...
00:14:29.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.519977 29728 catalog_manager.cc:6092] T 00000000000000000000000000000000 P b849676f6ae3499e801495a62dc3b393: Generated new TSK 0
00:14:29.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.522012 29728 catalog_manager.cc:1550] Initializing in-progress tserver states...
00:14:29.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.568976 29415 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.157.60" port: 33393 } attempt: 7
00:14:29.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.569633 29300 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu
00:14:29.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu
00:14:29.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/wal
00:14:29.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/data
00:14:29.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
00:14:29.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
00:14:29.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
00:14:29.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
00:14:29.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
00:14:29.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
00:14:29.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/logs
00:14:29.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/data/info.pb
00:14:29.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
00:14:29.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
00:14:29.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
00:14:29.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
00:14:29.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
00:14:29.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
00:14:29.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
00:14:29.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
00:14:29.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
00:14:29.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
00:14:29.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.28.157.1:0
00:14:29.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.28.157.1
00:14:29.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.28.157.1
00:14:29.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
00:14:29.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.28.157.62:35965,127.28.157.61:44801,127.28.157.60:33393
00:14:29.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.28.157.20:34671
00:14:29.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
00:14:29.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
00:14:29.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
00:14:29.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.604384 29415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22c0f0f5cbfc4ad683b142e008a47923: Bootstrap starting.
00:14:29.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.613991 29415 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22c0f0f5cbfc4ad683b142e008a47923: Neither blocks nor log segments found. Creating new log.
00:14:29.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.616603 29415 log.cc:826] T 00000000000000000000000000000000 P 22c0f0f5cbfc4ad683b142e008a47923: Log is configured to *not* fsync() on all Append() calls
00:14:29.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.635249 29415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22c0f0f5cbfc4ad683b142e008a47923: No bootstrap required, opened a new log
00:14:29.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.695003 29415 raft_consensus.cc:359] T 00000000000000000000000000000000 P 22c0f0f5cbfc4ad683b142e008a47923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22c0f0f5cbfc4ad683b142e008a47923" member_type: VOTER last_known_addr { host: "127.28.157.62" port: 35965 } } peers { permanent_uuid: "011d9300ff864ff9b52a082026a4136a" member_type: VOTER last_known_addr { host: "127.28.157.61" port: 44801 } } peers { permanent_uuid: "b849676f6ae3499e801495a62dc3b393" member_type: VOTER last_known_addr { host: "127.28.157.60" port: 33393 } }
00:14:29.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.697350 29415 raft_consensus.cc:385] T 00000000000000000000000000000000 P 22c0f0f5cbfc4ad683b142e008a47923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
00:14:29.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.698182 29415 raft_consensus.cc:740] T 00000000000000000000000000000000 P 22c0f0f5cbfc4ad683b142e008a47923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22c0f0f5cbfc4ad683b142e008a47923, State: Initialized, Role: FOLLOWER
00:14:29.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.700109 29415 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22c0f0f5cbfc4ad683b142e008a47923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22c0f0f5cbfc4ad683b142e008a47923" member_type: VOTER last_known_addr { host: "127.28.157.62" port: 35965 } } peers { permanent_uuid: "011d9300ff864ff9b52a082026a4136a" member_type: VOTER last_known_addr { host: "127.28.157.61" port: 44801 } } peers { permanent_uuid: "b849676f6ae3499e801495a62dc3b393" member_type: VOTER last_known_addr { host: "127.28.157.60" port: 33393 } }
00:14:29.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.724263 29415 sys_catalog.cc:565] T 00000000000000000000000000000000 P 22c0f0f5cbfc4ad683b142e008a47923 [sys.catalog]: configured and running, proceeding with master startup.
00:14:29.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.726323 29375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b849676f6ae3499e801495a62dc3b393" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22c0f0f5cbfc4ad683b142e008a47923"
00:14:29.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.724498 29379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b849676f6ae3499e801495a62dc3b393" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22c0f0f5cbfc4ad683b142e008a47923" is_pre_election: true
00:14:29.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.725778 29373 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 22c0f0f5cbfc4ad683b142e008a47923 [term 0 FOLLOWER]: Advancing to term 1
00:14:29.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.725630 29761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22c0f0f5cbfc4ad683b142e008a47923 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22c0f0f5cbfc4ad683b142e008a47923" member_type: VOTER last_known_addr { host: "127.28.157.62" port: 35965 } } peers { permanent_uuid: "011d9300ff864ff9b52a082026a4136a" member_type: VOTER last_known_addr { host: "127.28.157.61" port: 44801 } } peers { permanent_uuid: "b849676f6ae3499e801495a62dc3b393" member_type: VOTER last_known_addr { host: "127.28.157.60" port: 33393 } } }
00:14:29.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.731189 29761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22c0f0f5cbfc4ad683b142e008a47923 [sys.catalog]: This master's current role is: FOLLOWER
00:14:29.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.737927 29373 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 22c0f0f5cbfc4ad683b142e008a47923 [term 1 FOLLOWER]: Refusing update from remote peer b849676f6ae3499e801495a62dc3b393: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
00:14:29.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.741451 29753 consensus_queue.cc:1048] T 00000000000000000000000000000000 P b849676f6ae3499e801495a62dc3b393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22c0f0f5cbfc4ad683b142e008a47923" member_type: VOTER last_known_addr { host: "127.28.157.62" port: 35965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
00:14:29.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:29.759164 29345 tablet.cc:2406] T 00000000000000000000000000000000 P 22c0f0f5cbfc4ad683b142e008a47923: Can't schedule compaction. Clean time has not been advanced past its initial value.
00:14:29.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.818514 29761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22c0f0f5cbfc4ad683b142e008a47923 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b849676f6ae3499e801495a62dc3b393. Latest consensus state: current_term: 1 leader_uuid: "b849676f6ae3499e801495a62dc3b393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22c0f0f5cbfc4ad683b142e008a47923" member_type: VOTER last_known_addr { host: "127.28.157.62" port: 35965 } } peers { permanent_uuid: "011d9300ff864ff9b52a082026a4136a" member_type: VOTER last_known_addr { host: "127.28.157.61" port: 44801 } } peers { permanent_uuid: "b849676f6ae3499e801495a62dc3b393" member_type: VOTER last_known_addr { host: "127.28.157.60" port: 33393 } } }
00:14:29.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.819346 29761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22c0f0f5cbfc4ad683b142e008a47923 [sys.catalog]: This master's current role is: FOLLOWER
00:14:29.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.821209 29770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22c0f0f5cbfc4ad683b142e008a47923 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b849676f6ae3499e801495a62dc3b393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22c0f0f5cbfc4ad683b142e008a47923" member_type: VOTER last_known_addr { host: "127.28.157.62" port: 35965 } } peers { permanent_uuid: "011d9300ff864ff9b52a082026a4136a" member_type: VOTER last_known_addr { host: "127.28.157.61" port: 44801 } } peers { permanent_uuid: "b849676f6ae3499e801495a62dc3b393" member_type: VOTER last_known_addr { host: "127.28.157.60" port: 33393 } } }
00:14:29.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.822242 29770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22c0f0f5cbfc4ad683b142e008a47923 [sys.catalog]: This master's current role is: FOLLOWER
00:14:29.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:29.903241 29782 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
00:14:30.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:30.169330 29753 sys_catalog.cc:455] T 00000000000000000000000000000000 P b849676f6ae3499e801495a62dc3b393 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b849676f6ae3499e801495a62dc3b393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22c0f0f5cbfc4ad683b142e008a47923" member_type: VOTER last_known_addr { host: "127.28.157.62" port: 35965 } } peers { permanent_uuid: "011d9300ff864ff9b52a082026a4136a" member_type: VOTER last_known_addr { host: "127.28.157.61" port: 44801 } } peers { permanent_uuid: "b849676f6ae3499e801495a62dc3b393" member_type: VOTER last_known_addr { host: "127.28.157.60" port: 33393 } } }
00:14:30.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:30.170687 29753 sys_catalog.cc:458] T 00000000000000000000000000000000 P b849676f6ae3499e801495a62dc3b393 [sys.catalog]: This master's current role is: LEADER
00:14:30.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:30.182246 29756 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
00:14:30.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:30.182814 29756 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
00:14:30.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:30.183439 29756 flags.cc:432] Enabled unsafe flag: --never_fsync=true
00:14:30.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:30.223595 29756 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
00:14:30.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:30.224615 29756 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.157.1
00:14:30.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:30.264865 29756 tablet_server_runner.cc:78] Tablet server non-default flags:
00:14:30.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/logs
00:14:30.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
00:14:30.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
00:14:30.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
00:14:30.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.28.157.20:34671
00:14:30.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
00:14:30.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
00:14:30.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/data
00:14:30.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/wal
00:14:30.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
00:14:30.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
00:14:30.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.28.157.1:0
00:14:30.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
00:14:30.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
00:14:30.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
00:14:30.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/data/info.pb
00:14:30.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.28.157.1
00:14:30.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
00:14:30.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.28.157.62:35965,127.28.157.61:44801,127.28.157.60:33393
00:14:30.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
00:14:30.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
00:14:30.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
00:14:30.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
00:14:30.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
00:14:30.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.28.157.1
00:14:30.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
00:14:30.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
00:14:30.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
00:14:30.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision f7ed5b56c096f54e2f926bf94ceb7fc44c0c1329
00:14:30.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
00:14:30.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 11 Aug 2026 23:55:14 UTC on e7f111948823
00:14:30.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 17639
00:14:30.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
00:14:30.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:30.266731 29756 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
00:14:30.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:30.268986 29756 file_cache.cc:504] Constructed file cache file cache with capacity 419430
00:14:30.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:30.285589 29809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
00:14:30.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:30.393079 29738 catalog_manager.cc:1295] Loaded cluster ID: d9ab6a5c52014b6a9f2ed508c8bca5a3
00:14:30.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:30.393368 29738 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 011d9300ff864ff9b52a082026a4136a: loading cluster ID for follower catalog manager: success
00:14:30.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:30.420583 29738 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 011d9300ff864ff9b52a082026a4136a: acquiring CA information for follower catalog manager: success
00:14:30.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:30.426326 29738 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 011d9300ff864ff9b52a082026a4136a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
00:14:30.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:30.921828 29782 catalog_manager.cc:1295] Loaded cluster ID: d9ab6a5c52014b6a9f2ed508c8bca5a3
00:14:30.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:30.922575 29782 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 22c0f0f5cbfc4ad683b142e008a47923: loading cluster ID for follower catalog manager: success
00:14:30.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:30.929935 29782 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 22c0f0f5cbfc4ad683b142e008a47923: acquiring CA information for follower catalog manager: success
00:14:30.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:30.946961 29782 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 22c0f0f5cbfc4ad683b142e008a47923: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
00:14:32.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:30.287922 29810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
00:14:32.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:31.687248 29806 debug-util.cc:398] Leaking SignalData structure 0x7b080002ede0 after lost signal to thread 29756
00:14:32.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:32.128314 29756 thread.cc:641] GCE (cloud detector) Time spent creating pthread: real 1.842s	user 0.534s	sys 0.703s
00:14:32.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:32.131399 29756 thread.cc:608] GCE (cloud detector) Time spent starting thread: real 1.845s	user 0.534s	sys 0.703s
00:14:32.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:32.142335 29756 server_base.cc:1061] running on GCE node
00:14:32.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:32.143899 29816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
00:14:32.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:32.145207 29756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
00:14:32.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:32.147897 29756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
00:14:32.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:32.149829 29756 hybrid_clock.cc:648] HybridClock initialized: now 1786493672149747 us; error 44 us; skew 500 ppm
00:14:32.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:32.151242 29756 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
00:14:32.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:32.158645 29756 webserver.cc:533] Webserver started at http://127.28.157.1:35519/ using document root <none> and password file <none>
00:14:32.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:32.160245 29756 fs_manager.cc:362] Metadata directory not provided
00:14:32.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:32.160506 29756 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
00:14:32.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:32.161057 29756 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
00:14:32.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:32.166198 29756 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/data/instance:
00:14:32.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "70a127fea97b464a89d03c5848cf6637"
00:14:32.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 00:14:32 on dist-test-slave-t7g5"
00:14:32.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:32.167936 29756 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/wal/instance:
00:14:32.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "70a127fea97b464a89d03c5848cf6637"
00:14:32.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 00:14:32 on dist-test-slave-t7g5"
00:14:32.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:32.176549 29756 fs_manager.cc:696] Time spent creating directory manager: real 0.008s	user 0.004s	sys 0.004s
00:14:32.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:32.184058 29824 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
00:14:32.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:32.185387 29756 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.003s	sys 0.003s
00:14:32.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:32.185783 29756 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/data,/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/wal
00:14:32.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "70a127fea97b464a89d03c5848cf6637"
00:14:32.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 00:14:32 on dist-test-slave-t7g5"
00:14:32.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:32.186177 29756 fs_report.cc:389] FS layout report
00:14:32.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
00:14:32.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/wal
00:14:32.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/wal
00:14:32.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/data/data
00:14:32.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
00:14:32.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
00:14:32.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
00:14:32.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
00:14:32.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
00:14:32.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
00:14:32.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
00:14:32.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
00:14:32.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
00:14:32.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
00:14:32.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
00:14:32.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:32.246186 29756 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
00:14:32.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:32.247895 29756 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
00:14:32.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:32.248451 29756 kserver.cc:163] Server-wide thread pool size limit: 3276
00:14:32.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:32.252130 29756 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
00:14:32.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:32.258587 29756 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
00:14:32.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:32.258960 29756 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
00:14:32.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:32.259270 29756 ts_tablet_manager.cc:616] Registered 0 tablets
00:14:32.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:32.259526 29756 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
00:14:32.446 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:32.446180 29756 rpc_server.cc:307] RPC server started. Bound to: 127.28.157.1:37179
00:14:32.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:32.446782 30003 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.157.1:37179 every 8 connection(s)
00:14:32.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:32.448832 29756 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/data/info.pb
00:14:32.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:32.453733 29300 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu as pid 29756
00:14:32.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:32.454221 29300 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/wal/instance
00:14:32.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:32.472548 29300 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu
00:14:32.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu
00:14:32.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-1/wal
00:14:32.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-1/data
00:14:32.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
00:14:32.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
00:14:32.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
00:14:32.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
00:14:32.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
00:14:32.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
00:14:32.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-1/logs
00:14:32.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-1/data/info.pb
00:14:32.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
00:14:32.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
00:14:32.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
00:14:32.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
00:14:32.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
00:14:32.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
00:14:32.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
00:14:32.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
00:14:32.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
00:14:32.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
00:14:32.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.28.157.2:0
00:14:32.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.28.157.2
00:14:32.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.28.157.2
00:14:32.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
00:14:32.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.28.157.62:35965,127.28.157.61:44801,127.28.157.60:33393
00:14:32.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.28.157.20:34671
00:14:32.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
00:14:32.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
00:14:32.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
00:14:32.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:32.527557 30005 heartbeater.cc:344] Connected to a master server at 127.28.157.60:33393
00:14:32.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:32.528095 30005 heartbeater.cc:461] Registering TS with master...
00:14:32.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:32.529538 30005 heartbeater.cc:507] Master 127.28.157.60:33393 requested a full tablet report, sending...
00:14:32.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:32.532775 30013 heartbeater.cc:344] Connected to a master server at 127.28.157.62:35965
00:14:32.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:32.533154 30013 heartbeater.cc:461] Registering TS with master...
00:14:32.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:32.534169 30013 heartbeater.cc:507] Master 127.28.157.62:35965 requested a full tablet report, sending...
00:14:32.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:32.534317 29608 ts_manager.cc:194] Registered new tserver with Master: 70a127fea97b464a89d03c5848cf6637 (127.28.157.1:37179)
00:14:32.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:32.535677 30006 heartbeater.cc:344] Connected to a master server at 127.28.157.61:44801
00:14:32.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:32.536048 30006 heartbeater.cc:461] Registering TS with master...
00:14:32.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:32.536952 30006 heartbeater.cc:507] Master 127.28.157.61:44801 requested a full tablet report, sending...
00:14:32.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:32.537288 29361 ts_manager.cc:194] Registered new tserver with Master: 70a127fea97b464a89d03c5848cf6637 (127.28.157.1:37179)
00:14:32.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:32.537567 29608 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.28.157.1:35415
00:14:32.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:32.540512 29485 ts_manager.cc:194] Registered new tserver with Master: 70a127fea97b464a89d03c5848cf6637 (127.28.157.1:37179)
00:14:32.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:32.897843 30022 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
00:14:32.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:32.898336 30022 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
00:14:32.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:32.898883 30022 flags.cc:432] Enabled unsafe flag: --never_fsync=true
00:14:32.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:32.933801 30022 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
00:14:32.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:32.935045 30022 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.157.2
00:14:32.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:32.973940 30022 tablet_server_runner.cc:78] Tablet server non-default flags:
00:14:32.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-1/logs
00:14:32.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
00:14:32.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
00:14:32.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
00:14:32.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.28.157.20:34671
00:14:32.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
00:14:32.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
00:14:32.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-1/data
00:14:32.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-1/wal
00:14:32.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
00:14:32.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
00:14:32.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.28.157.2:0
00:14:32.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
00:14:32.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
00:14:32.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
00:14:32.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-1/data/info.pb
00:14:32.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.28.157.2
00:14:32.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
00:14:32.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.28.157.62:35965,127.28.157.61:44801,127.28.157.60:33393
00:14:32.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
00:14:32.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
00:14:32.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
00:14:32.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
00:14:32.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
00:14:32.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.28.157.2
00:14:32.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
00:14:32.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
00:14:32.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
00:14:32.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision f7ed5b56c096f54e2f926bf94ceb7fc44c0c1329
00:14:32.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
00:14:32.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 11 Aug 2026 23:55:14 UTC on e7f111948823
00:14:32.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 17639
00:14:32.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
00:14:32.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:32.975394 30022 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
00:14:32.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:32.977743 30022 file_cache.cc:504] Constructed file cache file cache with capacity 419430
00:14:32.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:32.993945 30035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
00:14:33.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:33.544719 30005 heartbeater.cc:499] Master 127.28.157.60:33393 was elected leader, sending a full tablet report...
00:14:34.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:32.994622 30036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
00:14:34.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:34.361616 30022 thread.cc:641] GCE (cloud detector) Time spent creating pthread: real 1.369s	user 0.489s	sys 0.867s
00:14:34.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:34.362049 30022 thread.cc:608] GCE (cloud detector) Time spent starting thread: real 1.370s	user 0.491s	sys 0.867s
00:14:34.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:34.363994 30022 server_base.cc:1061] running on GCE node
00:14:34.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:34.365617 30052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
00:14:34.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:34.367069 30022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
00:14:34.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:34.370075 30022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
00:14:34.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:34.371833 30022 hybrid_clock.cc:648] HybridClock initialized: now 1786493674371781 us; error 41 us; skew 500 ppm
00:14:34.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:34.372973 30022 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
00:14:34.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:34.383507 30022 webserver.cc:533] Webserver started at http://127.28.157.2:40801/ using document root <none> and password file <none>
00:14:34.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:34.384836 30022 fs_manager.cc:362] Metadata directory not provided
00:14:34.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:34.385135 30022 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
00:14:34.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:34.385738 30022 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
00:14:34.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:34.392244 30022 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-1/data/instance:
00:14:34.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "b5d8d89741dd4568a4c7b210a4da72d2"
00:14:34.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 00:14:34 on dist-test-slave-t7g5"
00:14:34.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:34.394061 30022 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-1/wal/instance:
00:14:34.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "b5d8d89741dd4568a4c7b210a4da72d2"
00:14:34.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 00:14:34 on dist-test-slave-t7g5"
00:14:34.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:34.405565 30022 fs_manager.cc:696] Time spent creating directory manager: real 0.011s	user 0.010s	sys 0.000s
00:14:34.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:34.414050 30059 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
00:14:34.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:34.416356 30022 fs_manager.cc:730] Time spent opening block manager: real 0.006s	user 0.004s	sys 0.000s
00:14:34.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:34.417130 30022 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-1/data,/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-1/wal
00:14:34.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "b5d8d89741dd4568a4c7b210a4da72d2"
00:14:34.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 00:14:34 on dist-test-slave-t7g5"
00:14:34.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:34.417629 30022 fs_report.cc:389] FS layout report
00:14:34.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
00:14:34.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-1/wal
00:14:34.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-1/wal
00:14:34.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-1/data/data
00:14:34.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
00:14:34.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
00:14:34.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
00:14:34.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
00:14:34.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
00:14:34.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
00:14:34.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
00:14:34.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
00:14:34.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
00:14:34.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
00:14:34.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
00:14:34.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:34.507987 30022 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
00:14:34.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:34.509999 30022 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
00:14:34.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:34.510694 30022 kserver.cc:163] Server-wide thread pool size limit: 3276
00:14:34.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:34.513561 30022 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
00:14:34.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:34.518774 30022 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
00:14:34.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:34.519027 30022 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
00:14:34.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:34.519227 30022 ts_tablet_manager.cc:616] Registered 0 tablets
00:14:34.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:34.519351 30022 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
00:14:34.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:34.685842 30022 rpc_server.cc:307] RPC server started. Bound to: 127.28.157.2:45951
00:14:34.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:34.685945 30232 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.157.2:45951 every 8 connection(s)
00:14:34.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:34.688573 30022 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-1/data/info.pb
00:14:34.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:34.691919 29300 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu as pid 30022
00:14:34.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:34.692608 29300 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-1/wal/instance
00:14:34.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:34.714741 29300 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu
00:14:34.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu
00:14:34.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-2/wal
00:14:34.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-2/data
00:14:34.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
00:14:34.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
00:14:34.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
00:14:34.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
00:14:34.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
00:14:34.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
00:14:34.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-2/logs
00:14:34.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-2/data/info.pb
00:14:34.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
00:14:34.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
00:14:34.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
00:14:34.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
00:14:34.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
00:14:34.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
00:14:34.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
00:14:34.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
00:14:34.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
00:14:34.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
00:14:34.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.28.157.3:0
00:14:34.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.28.157.3
00:14:34.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.28.157.3
00:14:34.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
00:14:34.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.28.157.62:35965,127.28.157.61:44801,127.28.157.60:33393
00:14:34.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.28.157.20:34671
00:14:34.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
00:14:34.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
00:14:34.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
00:14:34.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:34.729257 30237 heartbeater.cc:344] Connected to a master server at 127.28.157.62:35965
00:14:34.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:34.729890 30237 heartbeater.cc:461] Registering TS with master...
00:14:34.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:34.731274 30237 heartbeater.cc:507] Master 127.28.157.62:35965 requested a full tablet report, sending...
00:14:34.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:34.729327 30234 heartbeater.cc:344] Connected to a master server at 127.28.157.60:33393
00:14:34.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:34.732074 30234 heartbeater.cc:461] Registering TS with master...
00:14:34.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:34.733398 30234 heartbeater.cc:507] Master 127.28.157.60:33393 requested a full tablet report, sending...
00:14:34.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:34.736774 29361 ts_manager.cc:194] Registered new tserver with Master: b5d8d89741dd4568a4c7b210a4da72d2 (127.28.157.2:45951)
00:14:34.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:34.739321 29608 ts_manager.cc:194] Registered new tserver with Master: b5d8d89741dd4568a4c7b210a4da72d2 (127.28.157.2:45951)
00:14:34.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:34.741494 29608 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.28.157.2:39061
00:14:34.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:34.743798 30236 heartbeater.cc:344] Connected to a master server at 127.28.157.61:44801
00:14:34.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:34.744179 30236 heartbeater.cc:461] Registering TS with master...
00:14:34.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:34.745826 30236 heartbeater.cc:507] Master 127.28.157.61:44801 requested a full tablet report, sending...
00:14:34.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:34.748714 29485 ts_manager.cc:194] Registered new tserver with Master: b5d8d89741dd4568a4c7b210a4da72d2 (127.28.157.2:45951)
00:14:35.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:35.099005 30256 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
00:14:35.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:35.099531 30256 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
00:14:35.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:35.100051 30256 flags.cc:432] Enabled unsafe flag: --never_fsync=true
00:14:35.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:35.136374 30256 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
00:14:35.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:35.137219 30256 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.157.3
00:14:35.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:35.176443 30256 tablet_server_runner.cc:78] Tablet server non-default flags:
00:14:35.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-2/logs
00:14:35.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
00:14:35.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
00:14:35.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
00:14:35.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.28.157.20:34671
00:14:35.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
00:14:35.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
00:14:35.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-2/data
00:14:35.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-2/wal
00:14:35.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
00:14:35.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
00:14:35.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.28.157.3:0
00:14:35.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
00:14:35.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
00:14:35.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
00:14:35.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-2/data/info.pb
00:14:35.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.28.157.3
00:14:35.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
00:14:35.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.28.157.62:35965,127.28.157.61:44801,127.28.157.60:33393
00:14:35.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
00:14:35.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
00:14:35.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
00:14:35.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
00:14:35.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
00:14:35.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.28.157.3
00:14:35.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
00:14:35.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
00:14:35.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
00:14:35.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision f7ed5b56c096f54e2f926bf94ceb7fc44c0c1329
00:14:35.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
00:14:35.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 11 Aug 2026 23:55:14 UTC on e7f111948823
00:14:35.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 17639
00:14:35.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
00:14:35.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:35.178102 30256 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
00:14:35.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:35.180389 30256 file_cache.cc:504] Constructed file cache file cache with capacity 419430
00:14:35.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:35.198364 30269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
00:14:35.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:35.746901 30234 heartbeater.cc:499] Master 127.28.157.60:33393 was elected leader, sending a full tablet report...
00:14:36.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:35.201531 30270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
00:14:36.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:36.509754 30273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
00:14:36.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:36.513443 30272 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1307 milliseconds
00:14:36.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:36.513482 30256 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.313s	user 0.463s	sys 0.822s
00:14:36.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:36.513866 30256 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.314s	user 0.467s	sys 0.822s
00:14:36.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:36.514161 30256 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
00:14:36.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:36.515244 30256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
00:14:36.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:36.517675 30256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
00:14:36.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:36.519100 30256 hybrid_clock.cc:648] HybridClock initialized: now 1786493676519059 us; error 54 us; skew 500 ppm
00:14:36.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:36.520104 30256 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
00:14:36.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:36.528501 30256 webserver.cc:533] Webserver started at http://127.28.157.3:36411/ using document root <none> and password file <none>
00:14:36.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:36.529882 30256 fs_manager.cc:362] Metadata directory not provided
00:14:36.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:36.530128 30256 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
00:14:36.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:36.530867 30256 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
00:14:36.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:36.535645 30256 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-2/data/instance:
00:14:36.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "094dbc44c6eb439083946b5ab0085ad0"
00:14:36.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 00:14:36 on dist-test-slave-t7g5"
00:14:36.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:36.536909 30256 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-2/wal/instance:
00:14:36.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "094dbc44c6eb439083946b5ab0085ad0"
00:14:36.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 00:14:36 on dist-test-slave-t7g5"
00:14:36.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:36.545882 30256 fs_manager.cc:696] Time spent creating directory manager: real 0.008s	user 0.008s	sys 0.003s
00:14:36.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:36.552558 30284 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
00:14:36.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:36.553799 30256 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.005s	sys 0.001s
00:14:36.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:36.554108 30256 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-2/data,/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-2/wal
00:14:36.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "094dbc44c6eb439083946b5ab0085ad0"
00:14:36.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 00:14:36 on dist-test-slave-t7g5"
00:14:36.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:36.554414 30256 fs_report.cc:389] FS layout report
00:14:36.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
00:14:36.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-2/wal
00:14:36.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-2/wal
00:14:36.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-2/data/data
00:14:36.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
00:14:36.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
00:14:36.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
00:14:36.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
00:14:36.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
00:14:36.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
00:14:36.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
00:14:36.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
00:14:36.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
00:14:36.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
00:14:36.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
00:14:36.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:36.631498 30256 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
00:14:36.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:36.633066 30256 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
00:14:36.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:36.633505 30256 kserver.cc:163] Server-wide thread pool size limit: 3276
00:14:36.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:36.636417 30256 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
00:14:36.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:36.641116 30256 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
00:14:36.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:36.641356 30256 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
00:14:36.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:36.641642 30256 ts_tablet_manager.cc:616] Registered 0 tablets
00:14:36.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:36.641861 30256 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.001s	sys 0.000s
00:14:36.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:36.826766 30256 rpc_server.cc:307] RPC server started. Bound to: 127.28.157.3:42367
00:14:36.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:36.826829 30467 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.157.3:42367 every 8 connection(s)
00:14:36.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:36.830461 30256 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-2/data/info.pb
00:14:36.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:36.840373 29300 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu as pid 30256
00:14:36.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:36.840891 29300 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-2/wal/instance
00:14:36.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:36.893306 30470 heartbeater.cc:344] Connected to a master server at 127.28.157.61:44801
00:14:36.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:36.894133 30470 heartbeater.cc:461] Registering TS with master...
00:14:36.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:36.896152 30470 heartbeater.cc:507] Master 127.28.157.61:44801 requested a full tablet report, sending...
00:14:36.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:36.899190 30471 heartbeater.cc:344] Connected to a master server at 127.28.157.62:35965
00:14:36.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:36.900137 30471 heartbeater.cc:461] Registering TS with master...
00:14:36.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:36.900427 30469 heartbeater.cc:344] Connected to a master server at 127.28.157.60:33393
00:14:36.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:36.900869 30469 heartbeater.cc:461] Registering TS with master...
00:14:36.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:36.901284 30471 heartbeater.cc:507] Master 127.28.157.62:35965 requested a full tablet report, sending...
00:14:36.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:36.901589 29485 ts_manager.cc:194] Registered new tserver with Master: 094dbc44c6eb439083946b5ab0085ad0 (127.28.157.3:42367)
00:14:36.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:36.902027 30469 heartbeater.cc:507] Master 127.28.157.60:33393 requested a full tablet report, sending...
00:14:36.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:36.904286 29360 ts_manager.cc:194] Registered new tserver with Master: 094dbc44c6eb439083946b5ab0085ad0 (127.28.157.3:42367)
00:14:36.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:36.904570 29608 ts_manager.cc:194] Registered new tserver with Master: 094dbc44c6eb439083946b5ab0085ad0 (127.28.157.3:42367)
00:14:36.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:36.906081 29608 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.28.157.3:48765
00:14:36.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:36.924760 29300 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
00:14:36.937 [DEBUG - main] (MiniKuduCluster.java:195) Response: 
00:14:36.939 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_masters {
}

00:14:36.948 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_masters {
  masters {
    id {
      type: MASTER
      index: 0
    }
    bound_rpc_address {
      host: "127.28.157.62"
      port: 35965
    }
    bound_http_address {
      host: "127.28.157.62"
      port: 37877
    }
  }
  masters {
    id {
      type: MASTER
      index: 1
    }
    bound_rpc_address {
      host: "127.28.157.61"
      port: 44801
    }
    bound_http_address {
      host: "127.28.157.61"
      port: 41039
    }
  }
  masters {
    id {
      type: MASTER
      index: 2
    }
    bound_rpc_address {
      host: "127.28.157.60"
      port: 33393
    }
    bound_http_address {
      host: "127.28.157.60"
      port: 37633
    }
  }
}

00:14:36.951 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_tservers {
}

00:14:36.954 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_tservers {
  tservers {
    id {
      type: TSERVER
      index: 0
    }
    bound_rpc_address {
      host: "127.28.157.1"
      port: 37179
    }
    bound_http_address {
      host: "127.28.157.1"
      port: 35519
    }
  }
  tservers {
    id {
      type: TSERVER
      index: 1
    }
    bound_rpc_address {
      host: "127.28.157.2"
      port: 45951
    }
    bound_http_address {
      host: "127.28.157.2"
      port: 40801
    }
  }
  tservers {
    id {
      type: TSERVER
      index: 2
    }
    bound_rpc_address {
      host: "127.28.157.3"
      port: 42367
    }
    bound_http_address {
      host: "127.28.157.3"
      port: 36411
    }
  }
}

00:14:36.954 [INFO - main] (KuduTestHarness.java:157) Creating a new Kudu client...
00:14:37.153 [DEBUG - main] (CompositeStrategy.java:59) Unable to load modern Subject methods, relying only on legacy methods
java.lang.NoSuchMethodException: javax.security.auth.Subject.current()
	at java.base/java.lang.Class.getDeclaredMethod(Class.java:2677)
	at org.apache.kudu.client.internals.ModernStrategy.<init>(ModernStrategy.java:51)
	at org.apache.kudu.client.internals.CompositeStrategy.<init>(CompositeStrategy.java:52)
	at org.apache.kudu.client.internals.CompositeStrategy.<clinit>(CompositeStrategy.java:41)
	at org.apache.kudu.client.internals.SecurityManagerCompatibility.get(SecurityManagerCompatibility.java:43)
	at org.apache.kudu.client.SecurityContext.setupSubject(SecurityContext.java:155)
	at org.apache.kudu.client.SecurityContext.<init>(SecurityContext.java:138)
	at org.apache.kudu.client.AsyncKuduClient.<init>(AsyncKuduClient.java:431)
	at org.apache.kudu.client.AsyncKuduClient$AsyncKuduClientBuilder.build(AsyncKuduClient.java:3109)
	at org.apache.kudu.test.KuduTestHarness.before(KuduTestHarness.java:161)
	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:50)
	at org.apache.kudu.test.junit.RetryRule$RetryStatement.doOneAttempt(RetryRule.java:225)
	at org.apache.kudu.test.junit.RetryRule$RetryStatement.evaluate(RetryRule.java:246)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.junit.runners.Suite.runChild(Suite.java:128)
	at org.junit.runners.Suite.runChild(Suite.java:27)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
	at org.junit.runner.JUnitCore.runMain(JUnitCore.java:77)
	at org.junit.runner.JUnitCore.main(JUnitCore.java:36)
00:14:37.221 [DEBUG - main] (SecurityUtil.java:122) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 
00:14:37.422 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.28.157.62:35965(127.28.157.62:35965)] connecting to peer
00:14:37.480 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.28.157.61:44801(127.28.157.61:44801)] connecting to peer
00:14:37.482 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.28.157.60:33393(127.28.157.60:33393)] connecting to peer
00:14:37.502 [DEBUG - kudu-nio-2] (Connection.java:757) [peer master-127.28.157.60:33393(127.28.157.60:33393)] Successfully connected to peer
00:14:37.504 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.28.157.62:35965(127.28.157.62:35965)] Successfully connected to peer
00:14:37.510 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.28.157.61:44801(127.28.157.61:44801)] Successfully connected to peer
00:14:37.610 [DEBUG - kudu-nio-0] (Negotiator.java:528) SASL mechanism PLAIN chosen for peer 127.28.157.62
00:14:37.610 [DEBUG - kudu-nio-1] (Negotiator.java:528) SASL mechanism PLAIN chosen for peer 127.28.157.61
00:14:37.610 [DEBUG - kudu-nio-2] (Negotiator.java:528) SASL mechanism PLAIN chosen for peer 127.28.157.60
00:14:37.897 [DEBUG - kudu-nio-1] (Negotiator.java:918) Authenticated connection [id: 0xa3fc8b8d, L:/127.0.0.1:43862 - R:/127.28.157.61:44801] using SASL/PLAIN
00:14:37.897 [DEBUG - kudu-nio-2] (Negotiator.java:918) Authenticated connection [id: 0x2ecdc0bd, L:/127.0.0.1:59086 - R:/127.28.157.60:33393] using SASL/PLAIN
00:14:37.897 [DEBUG - kudu-nio-0] (Negotiator.java:918) Authenticated connection [id: 0xa370d54d, L:/127.0.0.1:42700 - R:/127.28.157.62:35965] using SASL/PLAIN
00:14:37.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:37.909807 30469 heartbeater.cc:499] Master 127.28.157.60:33393 was elected leader, sending a full tablet report...
00:14:37.940 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet Kudu Master for table 'Kudu Master' with partition [<start>, <end>)
00:14:37.941 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=CreateTable, tablet=null, attempt=1, TimeoutTracker(timeout=50000, elapsed=570), Traces: [0ms] refreshing cache from master, [33ms] Sub RPC ConnectToMaster: sending RPC to server master-127.28.157.62:35965, [105ms] Sub RPC ConnectToMaster: sending RPC to server master-127.28.157.61:44801, [107ms] Sub RPC ConnectToMaster: sending RPC to server master-127.28.157.60:33393, [542ms] Sub RPC ConnectToMaster: received response from server master-127.28.157.61:44801: OK, [542ms] Sub RPC ConnectToMaster: received response from server master-127.28.157.62:35965: OK, [542ms] Sub RPC ConnectToMaster: received response from server master-127.28.157.60:33393: OK, deferred=null) after lookup
00:14:37.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:37.991623 29608 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59086:
00:14:37.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) name: "org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353"
00:14:37.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) schema {
00:14:37.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
00:14:37.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "key"
00:14:37.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
00:14:37.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: true
00:14:37.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
00:14:37.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
00:14:37.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
00:14:37.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
00:14:37.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
00:14:37.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
00:14:37.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column1_i"
00:14:37.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
00:14:37.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
00:14:37.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
00:14:37.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
00:14:37.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
00:14:37.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
00:14:37.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
00:14:37.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
00:14:37.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column2_i"
00:14:37.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
00:14:37.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
00:14:37.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
00:14:37.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
00:14:37.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
00:14:37.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
00:14:37.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
00:14:37.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
00:14:37.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column3_s"
00:14:37.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: STRING
00:14:37.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
00:14:37.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: true
00:14:37.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     encoding: DICT_ENCODING
00:14:37.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     compression: LZ4
00:14:37.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 4096
00:14:37.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
00:14:37.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
00:14:37.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
00:14:37.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
00:14:37.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column4_b"
00:14:37.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: BOOL
00:14:37.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
00:14:37.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
00:14:37.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
00:14:37.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
00:14:37.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
00:14:37.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
00:14:37.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) }
00:14:37.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) partition_schema {
00:14:37.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   range_schema {
00:14:37.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     columns {
00:14:37.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)       name: "key"
00:14:37.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     }
00:14:37.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
00:14:37.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) }
00:14:37.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:37.998445 29608 catalog_manager.cc:7099] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
00:14:38.078 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:894) Opened table 879525e8d2a44cdca446658f4a03fc8d
00:14:38.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.125710 30132 tablet_service.cc:1511] Processing CreateTablet for tablet 2f4c1ce5aa4a4c8da912065e2decef6a (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353 [id=879525e8d2a44cdca446658f4a03fc8d]), partition=RANGE (key) PARTITION UNBOUNDED
00:14:38.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.125922 30366 tablet_service.cc:1511] Processing CreateTablet for tablet 2f4c1ce5aa4a4c8da912065e2decef6a (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353 [id=879525e8d2a44cdca446658f4a03fc8d]), partition=RANGE (key) PARTITION UNBOUNDED
00:14:38.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.128089 30132 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f4c1ce5aa4a4c8da912065e2decef6a. 1 dirs total, 0 dirs full, 0 dirs failed
00:14:38.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.128266 30366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f4c1ce5aa4a4c8da912065e2decef6a. 1 dirs total, 0 dirs full, 0 dirs failed
00:14:38.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.130507 29905 tablet_service.cc:1511] Processing CreateTablet for tablet 2f4c1ce5aa4a4c8da912065e2decef6a (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353 [id=879525e8d2a44cdca446658f4a03fc8d]), partition=RANGE (key) PARTITION UNBOUNDED
00:14:38.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.132663 29905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f4c1ce5aa4a4c8da912065e2decef6a. 1 dirs total, 0 dirs full, 0 dirs failed
00:14:38.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.158327 30536 tablet_bootstrap.cc:492] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2: Bootstrap starting.
00:14:38.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.163590 30540 tablet_bootstrap.cc:492] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637: Bootstrap starting.
00:14:38.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.164835 30537 tablet_bootstrap.cc:492] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0: Bootstrap starting.
00:14:38.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.167001 30536 tablet_bootstrap.cc:654] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2: Neither blocks nor log segments found. Creating new log.
00:14:38.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.169373 30536 log.cc:826] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2: Log is configured to *not* fsync() on all Append() calls
00:14:38.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.170477 30540 tablet_bootstrap.cc:654] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637: Neither blocks nor log segments found. Creating new log.
00:14:38.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.172241 30537 tablet_bootstrap.cc:654] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0: Neither blocks nor log segments found. Creating new log.
00:14:38.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.172946 30540 log.cc:826] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637: Log is configured to *not* fsync() on all Append() calls
00:14:38.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.175658 30537 log.cc:826] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0: Log is configured to *not* fsync() on all Append() calls
00:14:38.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.180130 30536 tablet_bootstrap.cc:492] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2: No bootstrap required, opened a new log
00:14:38.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.180986 30536 ts_tablet_manager.cc:1403] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2: Time spent bootstrapping tablet: real 0.027s	user 0.011s	sys 0.007s
00:14:38.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.182945 30540 tablet_bootstrap.cc:492] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637: No bootstrap required, opened a new log
00:14:38.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.183405 30540 ts_tablet_manager.cc:1403] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637: Time spent bootstrapping tablet: real 0.020s	user 0.004s	sys 0.011s
00:14:38.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.184291 30537 tablet_bootstrap.cc:492] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0: No bootstrap required, opened a new log
00:14:38.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.185369 30537 ts_tablet_manager.cc:1403] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0: Time spent bootstrapping tablet: real 0.021s	user 0.008s	sys 0.008s
00:14:38.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.208735 30540 raft_consensus.cc:359] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:14:38.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.209643 30540 raft_consensus.cc:385] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
00:14:38.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.209965 30540 raft_consensus.cc:740] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70a127fea97b464a89d03c5848cf6637, State: Initialized, Role: FOLLOWER
00:14:38.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.211143 30540 consensus_queue.cc:260] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:14:38.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.211382 30536 raft_consensus.cc:359] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:14:38.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.212224 30536 raft_consensus.cc:385] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
00:14:38.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.212531 30536 raft_consensus.cc:740] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5d8d89741dd4568a4c7b210a4da72d2, State: Initialized, Role: FOLLOWER
00:14:38.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.214133 30536 consensus_queue.cc:260] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:14:38.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.223340 30536 ts_tablet_manager.cc:1434] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2: Time spent starting tablet: real 0.042s	user 0.026s	sys 0.013s
00:14:38.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.224907 30540 ts_tablet_manager.cc:1434] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637: Time spent starting tablet: real 0.041s	user 0.026s	sys 0.012s
00:14:38.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.220384 30537 raft_consensus.cc:359] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:14:38.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.226554 30537 raft_consensus.cc:385] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
00:14:38.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.227046 30537 raft_consensus.cc:740] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 094dbc44c6eb439083946b5ab0085ad0, State: Initialized, Role: FOLLOWER
00:14:38.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.228343 30537 consensus_queue.cc:260] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:14:38.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.242985 30537 ts_tablet_manager.cc:1434] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0: Time spent starting tablet: real 0.057s	user 0.023s	sys 0.023s
00:14:38.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:38.272614 30017 tablet.cc:2406] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637: Can't schedule compaction. Clean time has not been advanced past its initial value.
00:14:38.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.300186 30547 raft_consensus.cc:493] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
00:14:38.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.300738 30547 raft_consensus.cc:515] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:14:38.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.303471 30547 leader_election.cc:290] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 094dbc44c6eb439083946b5ab0085ad0 (127.28.157.3:42367), 70a127fea97b464a89d03c5848cf6637 (127.28.157.1:37179)
00:14:38.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.317021 30404 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f4c1ce5aa4a4c8da912065e2decef6a" candidate_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "094dbc44c6eb439083946b5ab0085ad0" is_pre_election: true
00:14:38.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.317013 29934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f4c1ce5aa4a4c8da912065e2decef6a" candidate_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70a127fea97b464a89d03c5848cf6637" is_pre_election: true
00:14:38.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.317809 29934 raft_consensus.cc:2468] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5d8d89741dd4568a4c7b210a4da72d2 in term 0.
00:14:38.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.317836 30404 raft_consensus.cc:2468] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5d8d89741dd4568a4c7b210a4da72d2 in term 0.
00:14:38.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.319130 30060 leader_election.cc:304] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70a127fea97b464a89d03c5848cf6637, b5d8d89741dd4568a4c7b210a4da72d2; no voters: 
00:14:38.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.319825 30547 raft_consensus.cc:2804] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
00:14:38.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.320155 30547 raft_consensus.cc:493] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
00:14:38.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.320382 30547 raft_consensus.cc:3060] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [term 0 FOLLOWER]: Advancing to term 1
00:14:38.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.324795 30547 raft_consensus.cc:515] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:14:38.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.326251 30547 leader_election.cc:290] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [CANDIDATE]: Term 1 election: Requested vote from peers 094dbc44c6eb439083946b5ab0085ad0 (127.28.157.3:42367), 70a127fea97b464a89d03c5848cf6637 (127.28.157.1:37179)
00:14:38.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.327099 30404 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f4c1ce5aa4a4c8da912065e2decef6a" candidate_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "094dbc44c6eb439083946b5ab0085ad0"
00:14:38.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.327245 29934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f4c1ce5aa4a4c8da912065e2decef6a" candidate_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70a127fea97b464a89d03c5848cf6637"
00:14:38.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.327540 30404 raft_consensus.cc:3060] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 0 FOLLOWER]: Advancing to term 1
00:14:38.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.327666 29934 raft_consensus.cc:3060] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [term 0 FOLLOWER]: Advancing to term 1
00:14:38.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.332214 29934 raft_consensus.cc:2468] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5d8d89741dd4568a4c7b210a4da72d2 in term 1.
00:14:38.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.332357 30404 raft_consensus.cc:2468] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5d8d89741dd4568a4c7b210a4da72d2 in term 1.
00:14:38.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.333112 30060 leader_election.cc:304] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70a127fea97b464a89d03c5848cf6637, b5d8d89741dd4568a4c7b210a4da72d2; no voters: 
00:14:38.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.333724 30547 raft_consensus.cc:2804] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [term 1 FOLLOWER]: Leader election won for term 1
00:14:38.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.335264 30547 raft_consensus.cc:697] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [term 1 LEADER]: Becoming Leader. State: Replica: b5d8d89741dd4568a4c7b210a4da72d2, State: Running, Role: LEADER
00:14:38.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.336349 30547 consensus_queue.cc:237] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:14:38.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.348083 29607 catalog_manager.cc:5719] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to b5d8d89741dd4568a4c7b210a4da72d2 (127.28.157.2). New cstate: current_term: 1 leader_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } health_report { overall_health: UNKNOWN } } }
00:14:38.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:38.373548 30480 tablet.cc:2406] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0: Can't schedule compaction. Clean time has not been advanced past its initial value.
00:14:38.401 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:894) Opened table 879525e8d2a44cdca446658f4a03fc8d
00:14:38.424 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:14:38.433 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299991, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367),70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179),b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)[L]]}]
00:14:38.434 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=null, attempt=1, sequence_id=-1, TimeoutTracker(timeout=30000, elapsed=23), Traces: [0ms] refreshing cache from master, [0ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [11ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=null) after lookup
00:14:38.445 [DEBUG - kudu-nio-2] (Connection.java:748) [peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)] connecting to peer
00:14:38.448 [DEBUG - kudu-nio-3] (Connection.java:757) [peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)] Successfully connected to peer
00:14:38.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:38.455325 30239 tablet.cc:2406] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
00:14:38.526 [DEBUG - kudu-nio-3] (Negotiator.java:918) Authenticated connection [id: 0xe74d3d22, L:/127.0.0.1:49378 - R:/127.28.157.2:45951] using TOKEN/null
00:14:38.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.541106 29934 raft_consensus.cc:1275] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [term 1 FOLLOWER]: Refusing update from remote peer b5d8d89741dd4568a4c7b210a4da72d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
00:14:38.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.541608 30404 raft_consensus.cc:1275] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 1 FOLLOWER]: Refusing update from remote peer b5d8d89741dd4568a4c7b210a4da72d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
00:14:38.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.542886 30551 consensus_queue.cc:1048] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
00:14:38.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.543691 30547 consensus_queue.cc:1048] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
00:14:38.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.574990 30558 mvcc.cc:204] Tried to move back new op lower bound from 7317478107287883776 to 7317478106474266624. Current Snapshot: MvccSnapshot[applied={T|T < 7317478107287883776}]
00:14:38.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.586894 30560 mvcc.cc:204] Tried to move back new op lower bound from 7317478107287883776 to 7317478106474266624. Current Snapshot: MvccSnapshot[applied={T|T < 7317478107287883776}]
00:14:38.829 [DEBUG - kudu-nio-3] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 3 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
00:14:38.831 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.28.157.2:45951
00:14:38.832 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

00:14:38.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:38.833503 29300 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu with pid 30022
00:14:38.932 [DEBUG - kudu-nio-3] (Connection.java:239) [peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)] handling channelInactive
00:14:38.932 [DEBUG - kudu-nio-3] (Connection.java:709) [peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)] cleaning up while in state READY due to: connection closed
00:14:38.935 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
00:14:38.936 [DEBUG - Time-limited test] (Connection.java:748) [peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)] connecting to peer
00:14:38.940 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951
00:14:38.940 [DEBUG - kudu-nio-4] (Connection.java:709) [peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)] cleaning up while in state CONNECTING due to: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951
00:14:38.941 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951
00:14:38.958 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:14:38.958 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367),70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179),b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)[L]]}]
00:14:38.959 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=2, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=23), Traces: [0ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [6ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [6ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [22ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@235472267(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@784c68f9 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@14ae596f -> wakeup thread Time-limited test)) after lookup
00:14:38.959 [DEBUG - kudu-nio-2] (Connection.java:748) [peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)] connecting to peer
00:14:38.961 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951
00:14:38.962 [DEBUG - kudu-nio-5] (Connection.java:709) [peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)] cleaning up while in state CONNECTING due to: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951
00:14:38.962 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951
00:14:38.979 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:14:38.980 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367),70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179),b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)[L]]}]
00:14:38.980 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=4, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=45), Traces: [0ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [6ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [6ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [22ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [24ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [27ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [27ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [37ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [44ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@235472267(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@784c68f9 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1223594241 after retry RPC@1261292494), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@14ae596f -> wakeup thread Time-limited test -> (continuation of Deferred@1223594241 after retry RPC@1261292494))) after lookup
00:14:38.981 [DEBUG - kudu-nio-2] (Connection.java:748) [peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)] connecting to peer
00:14:38.983 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951
00:14:38.984 [DEBUG - kudu-nio-6] (Connection.java:709) [peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)] cleaning up while in state CONNECTING due to: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951
00:14:38.984 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951
00:14:39.018 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:14:39.019 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367),70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179),b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)[L]]}]
00:14:39.019 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=6, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=84), Traces: [0ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [6ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [6ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [22ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [24ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [27ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [27ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [37ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [44ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [46ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [49ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [49ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [77ms] refreshing cache from master, [77ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [83ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@235472267(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@784c68f9 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1223594241 after retry RPC@1261292494) -> (continuation of Deferred@1279296026 after retry RPC@1901378313), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@14ae596f -> wakeup thread Time-limited test -> (continuation of Deferred@1223594241 after retry RPC@1261292494) -> (continuation of Deferred@1279296026 after retry RPC@1901378313))) after lookup
00:14:39.020 [DEBUG - kudu-nio-2] (Connection.java:748) [peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)] connecting to peer
00:14:39.022 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951
00:14:39.023 [DEBUG - kudu-nio-7] (Connection.java:709) [peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)] cleaning up while in state CONNECTING due to: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951
00:14:39.023 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951
00:14:39.098 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:14:39.098 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367),70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179),b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)[L]]}]
00:14:39.099 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=8, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=164), Traces: [0ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [6ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [6ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [22ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [24ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [27ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [27ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [37ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [44ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [46ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [49ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [49ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [77ms] refreshing cache from master, [77ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [83ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [85ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [88ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [88ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [156ms] refreshing cache from master, [157ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [163ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@235472267(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@784c68f9 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1223594241 after retry RPC@1261292494) -> (continuation of Deferred@1279296026 after retry RPC@1901378313) -> (continuation of Deferred@109856617 after retry RPC@451669093), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@14ae596f -> wakeup thread Time-limited test -> (continuation of Deferred@1223594241 after retry RPC@1261292494) -> (continuation of Deferred@1279296026 after retry RPC@1901378313) -> (continuation of Deferred@109856617 after retry RPC@451669093))) after lookup
00:14:39.100 [DEBUG - kudu-nio-2] (Connection.java:748) [peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)] connecting to peer
00:14:39.101 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951
00:14:39.102 [DEBUG - kudu-nio-0] (Connection.java:709) [peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)] cleaning up while in state CONNECTING due to: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951
00:14:39.102 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951
00:14:39.498 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:14:39.499 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367),70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179),b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)[L]]}]
00:14:39.499 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=10, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=564), Traces: [0ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [6ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [6ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [22ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [24ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [27ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [27ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [37ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [44ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [46ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [49ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [49ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [77ms] refreshing cache from master, [77ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [83ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [85ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [88ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [88ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [156ms] refreshing cache from master, [157ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [163ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [165ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [167ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [167ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [557ms] refreshing cache from master, [558ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [562ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@235472267(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@784c68f9 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1223594241 after retry RPC@1261292494) -> (continuation of Deferred@1279296026 after retry RPC@1901378313) -> (continuation of Deferred@109856617 after retry RPC@451669093) -> (continuation of Deferred@1981829280 after retry RPC@871170757), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@14ae596f -> wakeup thread Time-limited test -> (continuation of Deferred@1223594241 after retry RPC@1261292494) -> (continuation of Deferred@1279296026 after retry RPC@1901378313) -> (continuation of Deferred@109856617 after retry RPC@451669093) -> (continuation of Deferred@1981829280 after retry RPC@871170757))) after lookup
00:14:39.500 [DEBUG - kudu-nio-2] (Connection.java:748) [peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)] connecting to peer
00:14:39.502 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951
00:14:39.502 [DEBUG - kudu-nio-1] (Connection.java:709) [peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)] cleaning up while in state CONNECTING due to: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951
00:14:39.502 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951
00:14:40.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:40.199429 30583 raft_consensus.cc:493] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b5d8d89741dd4568a4c7b210a4da72d2)
00:14:40.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:40.200168 30583 raft_consensus.cc:515] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:14:40.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:40.203351 30583 leader_election.cc:290] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b5d8d89741dd4568a4c7b210a4da72d2 (127.28.157.2:45951), 70a127fea97b464a89d03c5848cf6637 (127.28.157.1:37179)
00:14:40.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:40.207748 30289 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.157.2:45951: connect: Connection refused (error 111)
00:14:40.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:40.212579 30289 leader_election.cc:336] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b5d8d89741dd4568a4c7b210a4da72d2 (127.28.157.2:45951): Network error: Client connection negotiation failed: client connection to 127.28.157.2:45951: connect: Connection refused (error 111)
00:14:40.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:40.216773 29934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f4c1ce5aa4a4c8da912065e2decef6a" candidate_uuid: "094dbc44c6eb439083946b5ab0085ad0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "70a127fea97b464a89d03c5848cf6637" is_pre_election: true
00:14:40.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:40.219020 30285 leader_election.cc:304] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 094dbc44c6eb439083946b5ab0085ad0; no voters: 70a127fea97b464a89d03c5848cf6637, b5d8d89741dd4568a4c7b210a4da72d2
00:14:40.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:40.219928 30583 raft_consensus.cc:2749] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
00:14:40.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:40.233618 30589 raft_consensus.cc:493] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b5d8d89741dd4568a4c7b210a4da72d2)
00:14:40.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:40.234318 30589 raft_consensus.cc:515] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:14:40.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:40.237323 30589 leader_election.cc:290] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b5d8d89741dd4568a4c7b210a4da72d2 (127.28.157.2:45951), 094dbc44c6eb439083946b5ab0085ad0 (127.28.157.3:42367)
00:14:40.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:40.242775 29830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.157.2:45951: connect: Connection refused (error 111)
00:14:40.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:40.247510 29830 leader_election.cc:336] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b5d8d89741dd4568a4c7b210a4da72d2 (127.28.157.2:45951): Network error: Client connection negotiation failed: client connection to 127.28.157.2:45951: connect: Connection refused (error 111)
00:14:40.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:40.252822 30404 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f4c1ce5aa4a4c8da912065e2decef6a" candidate_uuid: "70a127fea97b464a89d03c5848cf6637" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "094dbc44c6eb439083946b5ab0085ad0" is_pre_election: true
00:14:40.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:40.253319 30404 raft_consensus.cc:2468] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70a127fea97b464a89d03c5848cf6637 in term 1.
00:14:40.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:40.254492 29826 leader_election.cc:304] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 094dbc44c6eb439083946b5ab0085ad0, 70a127fea97b464a89d03c5848cf6637; no voters: b5d8d89741dd4568a4c7b210a4da72d2
00:14:40.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:40.255275 30589 raft_consensus.cc:2804] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [term 1 FOLLOWER]: Leader pre-election won for term 2
00:14:40.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:40.255599 30589 raft_consensus.cc:493] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b5d8d89741dd4568a4c7b210a4da72d2)
00:14:40.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:40.255918 30589 raft_consensus.cc:3060] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [term 1 FOLLOWER]: Advancing to term 2
00:14:40.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:40.261035 30589 raft_consensus.cc:515] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:14:40.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:40.262809 30589 leader_election.cc:290] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [CANDIDATE]: Term 2 election: Requested vote from peers b5d8d89741dd4568a4c7b210a4da72d2 (127.28.157.2:45951), 094dbc44c6eb439083946b5ab0085ad0 (127.28.157.3:42367)
00:14:40.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:40.264623 30404 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f4c1ce5aa4a4c8da912065e2decef6a" candidate_uuid: "70a127fea97b464a89d03c5848cf6637" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "094dbc44c6eb439083946b5ab0085ad0"
00:14:40.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:40.265300 30404 raft_consensus.cc:3060] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 1 FOLLOWER]: Advancing to term 2
00:14:40.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:40.267915 29830 leader_election.cc:336] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b5d8d89741dd4568a4c7b210a4da72d2 (127.28.157.2:45951): Network error: Client connection negotiation failed: client connection to 127.28.157.2:45951: connect: Connection refused (error 111)
00:14:40.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:40.270210 30404 raft_consensus.cc:2468] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70a127fea97b464a89d03c5848cf6637 in term 2.
00:14:40.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:40.271267 29826 leader_election.cc:304] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 094dbc44c6eb439083946b5ab0085ad0, 70a127fea97b464a89d03c5848cf6637; no voters: b5d8d89741dd4568a4c7b210a4da72d2
00:14:40.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:40.271951 30589 raft_consensus.cc:2804] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [term 2 FOLLOWER]: Leader election won for term 2
00:14:40.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:40.273684 30589 raft_consensus.cc:697] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [term 2 LEADER]: Becoming Leader. State: Replica: 70a127fea97b464a89d03c5848cf6637, State: Running, Role: LEADER
00:14:40.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:40.274888 30589 consensus_queue.cc:237] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:14:40.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:40.286522 29607 catalog_manager.cc:5719] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 reported cstate change: term changed from 1 to 2, leader changed from b5d8d89741dd4568a4c7b210a4da72d2 (127.28.157.2) to 70a127fea97b464a89d03c5848cf6637 (127.28.157.1). New cstate: current_term: 2 leader_uuid: "70a127fea97b464a89d03c5848cf6637" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } health_report { overall_health: HEALTHY } } }
00:14:40.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:40.794325 30404 raft_consensus.cc:1275] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 2 FOLLOWER]: Refusing update from remote peer 70a127fea97b464a89d03c5848cf6637: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
00:14:40.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:40.796087 30616 consensus_queue.cc:1048] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [LEADER]: Connected to new peer: Peer: permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
00:14:40.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:40.826676 29830 consensus_peers.cc:597] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 -> Peer b5d8d89741dd4568a4c7b210a4da72d2 (127.28.157.2:45951): Couldn't send request to peer b5d8d89741dd4568a4c7b210a4da72d2. Status: Network error: Client connection negotiation failed: client connection to 127.28.157.2:45951: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
00:14:41.037 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:14:41.038 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367),70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)[L],b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)]}]
00:14:41.039 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=12, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=2103), Traces: [0ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [6ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [6ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [22ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [24ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [27ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [27ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [37ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [44ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [46ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [49ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [49ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [77ms] refreshing cache from master, [77ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [83ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [85ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [88ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [88ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [156ms] refreshing cache from master, [157ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [163ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [165ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [167ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [167ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [557ms] refreshing cache from master, [558ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [562ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [565ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [567ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [567ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [2097ms] refreshing cache from master, [2097ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [2102ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@235472267(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@784c68f9 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1223594241 after retry RPC@1261292494) -> (continuation of Deferred@1279296026 after retry RPC@1901378313) -> (continuation of Deferred@109856617 after retry RPC@451669093) -> (continuation of Deferred@1981829280 after retry RPC@871170757) -> (continuation of Deferred@962009482 after retry RPC@1652023119), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@14ae596f -> wakeup thread Time-limited test -> (continuation of Deferred@1223594241 after retry RPC@1261292494) -> (continuation of Deferred@1279296026 after retry RPC@1901378313) -> (continuation of Deferred@109856617 after retry RPC@451669093) -> (continuation of Deferred@1981829280 after retry RPC@871170757) -> (continuation of Deferred@962009482 after retry RPC@1652023119))) after lookup
00:14:41.040 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] connecting to peer
00:14:41.042 [DEBUG - kudu-nio-2] (Connection.java:757) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] Successfully connected to peer
00:14:41.072 [DEBUG - kudu-nio-2] (Negotiator.java:918) Authenticated connection [id: 0xcb2dc656, L:/127.0.0.1:51966 - R:/127.28.157.1:37179] using TOKEN/null
00:14:41.122 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.28.157.2:45951
00:14:41.125 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

00:14:41.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:41.126573 29300 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu
00:14:41.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu
00:14:41.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-1/wal
00:14:41.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-1/data
00:14:41.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
00:14:41.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
00:14:41.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
00:14:41.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
00:14:41.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
00:14:41.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
00:14:41.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-1/logs
00:14:41.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-1/data/info.pb
00:14:41.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
00:14:41.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
00:14:41.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
00:14:41.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
00:14:41.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
00:14:41.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
00:14:41.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
00:14:41.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
00:14:41.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
00:14:41.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
00:14:41.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.28.157.2:45951
00:14:41.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.28.157.2
00:14:41.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.28.157.62:35965,127.28.157.61:44801,127.28.157.60:33393
00:14:41.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40801
00:14:41.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.28.157.2
00:14:41.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.28.157.20:34671
00:14:41.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
00:14:41.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
00:14:41.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
00:14:41.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:41.471923 30644 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
00:14:41.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:41.472487 30644 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
00:14:41.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:41.473105 30644 flags.cc:432] Enabled unsafe flag: --never_fsync=true
00:14:41.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:41.507071 30644 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
00:14:41.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:41.508114 30644 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.157.2
00:14:41.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:41.547680 30644 tablet_server_runner.cc:78] Tablet server non-default flags:
00:14:41.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-1/logs
00:14:41.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
00:14:41.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
00:14:41.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
00:14:41.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.28.157.20:34671
00:14:41.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
00:14:41.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
00:14:41.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-1/data
00:14:41.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-1/wal
00:14:41.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
00:14:41.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
00:14:41.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.28.157.2:45951
00:14:41.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
00:14:41.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
00:14:41.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
00:14:41.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-1/data/info.pb
00:14:41.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.28.157.2
00:14:41.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40801
00:14:41.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.28.157.62:35965,127.28.157.61:44801,127.28.157.60:33393
00:14:41.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
00:14:41.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
00:14:41.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
00:14:41.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
00:14:41.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
00:14:41.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.28.157.2
00:14:41.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
00:14:41.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
00:14:41.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
00:14:41.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision f7ed5b56c096f54e2f926bf94ceb7fc44c0c1329
00:14:41.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
00:14:41.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 11 Aug 2026 23:55:14 UTC on e7f111948823
00:14:41.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 17639
00:14:41.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
00:14:41.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:41.549474 30644 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
00:14:41.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:41.551550 30644 file_cache.cc:504] Constructed file cache file cache with capacity 419430
00:14:41.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:41.565996 30663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
00:14:43.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:42.968333 30662 debug-util.cc:398] Leaking SignalData structure 0x7b080002ede0 after lost signal to thread 30644
00:14:43.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:43.018967 30644 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.453s	user 0.513s	sys 0.851s
00:14:43.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:43.019392 30644 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.454s	user 0.513s	sys 0.851s
00:14:43.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:41.567157 30664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
00:14:43.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:43.021198 30668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
00:14:43.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:43.023976 30665 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1452 milliseconds
00:14:43.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:43.024044 30644 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
00:14:43.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:43.025220 30644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
00:14:43.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:43.028785 30644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
00:14:43.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:43.030360 30644 hybrid_clock.cc:648] HybridClock initialized: now 1786493683030301 us; error 76 us; skew 500 ppm
00:14:43.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:43.031235 30644 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
00:14:43.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:43.037577 30644 webserver.cc:533] Webserver started at http://127.28.157.2:40801/ using document root <none> and password file <none>
00:14:43.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:43.038779 30644 fs_manager.cc:362] Metadata directory not provided
00:14:43.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:43.039091 30644 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
00:14:43.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:43.047834 30644 fs_manager.cc:714] Time spent opening directory manager: real 0.005s	user 0.005s	sys 0.000s
00:14:43.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:43.053530 30681 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
00:14:43.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:43.055163 30644 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.003s	sys 0.001s
00:14:43.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:43.055626 30644 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-1/data,/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-1/wal
00:14:43.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "b5d8d89741dd4568a4c7b210a4da72d2"
00:14:43.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 00:14:34 on dist-test-slave-t7g5"
00:14:43.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:43.058368 30644 fs_report.cc:389] FS layout report
00:14:43.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
00:14:43.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-1/wal
00:14:43.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-1/wal
00:14:43.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-1/data/data
00:14:43.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
00:14:43.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
00:14:43.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
00:14:43.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
00:14:43.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
00:14:43.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
00:14:43.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
00:14:43.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
00:14:43.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
00:14:43.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
00:14:43.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
00:14:43.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:43.128903 30644 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
00:14:43.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:43.130486 30644 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
00:14:43.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:43.130903 30644 kserver.cc:163] Server-wide thread pool size limit: 3276
00:14:43.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:43.133523 30644 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
00:14:43.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:43.139921 30697 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
00:14:43.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:43.148984 30644 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
00:14:43.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:43.149356 30644 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.011s	user 0.002s	sys 0.000s
00:14:43.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:43.149685 30644 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
00:14:43.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:43.155208 30644 ts_tablet_manager.cc:616] Registered 1 tablets
00:14:43.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:43.155441 30644 ts_tablet_manager.cc:595] Time spent register tablets: real 0.006s	user 0.005s	sys 0.000s
00:14:43.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:43.156226 30697 tablet_bootstrap.cc:492] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2: Bootstrap starting.
00:14:43.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:43.217165 30697 log.cc:826] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2: Log is configured to *not* fsync() on all Append() calls
00:14:43.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:43.256654 30697 tablet_bootstrap.cc:492] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
00:14:43.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:43.257889 30697 tablet_bootstrap.cc:492] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2: Bootstrap complete.
00:14:43.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:43.259905 30697 ts_tablet_manager.cc:1403] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2: Time spent bootstrapping tablet: real 0.104s	user 0.068s	sys 0.030s
00:14:43.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:43.285615 30697 raft_consensus.cc:359] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:14:43.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:43.289240 30697 raft_consensus.cc:740] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5d8d89741dd4568a4c7b210a4da72d2, State: Initialized, Role: FOLLOWER
00:14:43.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:43.290518 30697 consensus_queue.cc:260] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:14:43.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:43.294539 30697 ts_tablet_manager.cc:1434] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2: Time spent starting tablet: real 0.034s	user 0.028s	sys 0.008s
00:14:43.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:43.330777 30644 rpc_server.cc:307] RPC server started. Bound to: 127.28.157.2:45951
00:14:43.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:43.331297 30874 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.157.2:45951 every 8 connection(s)
00:14:43.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:43.334901 30644 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-1/data/info.pb
00:14:43.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:43.337841 29300 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu as pid 30644
00:14:43.339 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
00:14:43.376 [DEBUG - kudu-nio-2] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 6 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
00:14:43.377 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.28.157.1:37179
00:14:43.377 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

00:14:43.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:43.378829 29300 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu with pid 29756
00:14:43.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:43.383574 30875 heartbeater.cc:344] Connected to a master server at 127.28.157.60:33393
00:14:43.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:43.384225 30875 heartbeater.cc:461] Registering TS with master...
00:14:43.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:43.385876 30875 heartbeater.cc:507] Master 127.28.157.60:33393 requested a full tablet report, sending...
00:14:43.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:43.391211 29608 ts_manager.cc:194] Re-registered known tserver with Master: b5d8d89741dd4568a4c7b210a4da72d2 (127.28.157.2:45951)
00:14:43.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:43.394488 29608 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.28.157.2:51897
00:14:43.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:43.402799 30876 heartbeater.cc:344] Connected to a master server at 127.28.157.61:44801
00:14:43.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:43.403209 30876 heartbeater.cc:461] Registering TS with master...
00:14:43.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:43.403873 30876 heartbeater.cc:507] Master 127.28.157.61:44801 requested a full tablet report, sending...
00:14:43.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:43.407507 29481 ts_manager.cc:194] Re-registered known tserver with Master: b5d8d89741dd4568a4c7b210a4da72d2 (127.28.157.2:45951)
00:14:43.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:43.410955 30883 heartbeater.cc:344] Connected to a master server at 127.28.157.62:35965
00:14:43.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:43.411334 30883 heartbeater.cc:461] Registering TS with master...
00:14:43.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:43.411969 30883 heartbeater.cc:507] Master 127.28.157.62:35965 requested a full tablet report, sending...
00:14:43.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:43.415774 29360 ts_manager.cc:194] Re-registered known tserver with Master: b5d8d89741dd4568a4c7b210a4da72d2 (127.28.157.2:45951)
00:14:43.429 [DEBUG - kudu-nio-2] (Connection.java:239) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] handling channelInactive
00:14:43.430 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] cleaning up while in state READY due to: connection closed
00:14:43.430 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
00:14:43.431 [DEBUG - Time-limited test] (Connection.java:748) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] connecting to peer
00:14:43.432 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:14:43.432 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] cleaning up while in state CONNECTING due to: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:14:43.432 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:14:43.457 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:14:43.457 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367),70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)[L],b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)]}]
00:14:43.458 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=2, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=27), Traces: [0ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [1ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [2ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [26ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@141882149(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@be0519d -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5d198624 -> wakeup thread Time-limited test)) after lookup
00:14:43.458 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] connecting to peer
00:14:43.459 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:14:43.460 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] cleaning up while in state CONNECTING due to: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:14:43.460 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:14:43.477 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:14:43.478 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367),70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)[L],b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)]}]
00:14:43.478 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=4, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=47), Traces: [0ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [1ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [2ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [26ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [27ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [29ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [29ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [41ms] refreshing cache from master, [41ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [46ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@141882149(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@be0519d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@128719096 after retry RPC@370513551), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5d198624 -> wakeup thread Time-limited test -> (continuation of Deferred@128719096 after retry RPC@370513551))) after lookup
00:14:43.479 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] connecting to peer
00:14:43.481 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:14:43.481 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] cleaning up while in state CONNECTING due to: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:14:43.482 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:14:43.497 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:14:43.498 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367),70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)[L],b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)]}]
00:14:43.498 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=6, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=67), Traces: [0ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [1ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [2ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [26ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [27ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [29ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [29ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [41ms] refreshing cache from master, [41ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [46ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [47ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [50ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [51ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [61ms] refreshing cache from master, [62ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [66ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@141882149(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@be0519d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@128719096 after retry RPC@370513551) -> (continuation of Deferred@1332949365 after retry RPC@1978169338), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5d198624 -> wakeup thread Time-limited test -> (continuation of Deferred@128719096 after retry RPC@370513551) -> (continuation of Deferred@1332949365 after retry RPC@1978169338))) after lookup
00:14:43.500 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] connecting to peer
00:14:43.501 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:14:43.501 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] cleaning up while in state CONNECTING due to: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:14:43.501 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:14:43.517 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:14:43.517 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367),70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)[L],b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)]}]
00:14:43.518 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=8, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=87), Traces: [0ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [1ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [2ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [26ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [27ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [29ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [29ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [41ms] refreshing cache from master, [41ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [46ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [47ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [50ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [51ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [61ms] refreshing cache from master, [62ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [66ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [68ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [70ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [70ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [81ms] refreshing cache from master, [81ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [85ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@141882149(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@be0519d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@128719096 after retry RPC@370513551) -> (continuation of Deferred@1332949365 after retry RPC@1978169338) -> (continuation of Deferred@757248130 after retry RPC@457393531), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5d198624 -> wakeup thread Time-limited test -> (continuation of Deferred@128719096 after retry RPC@370513551) -> (continuation of Deferred@1332949365 after retry RPC@1978169338) -> (continuation of Deferred@757248130 after retry RPC@457393531))) after lookup
00:14:43.519 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] connecting to peer
00:14:43.520 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:14:43.520 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] cleaning up while in state CONNECTING due to: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:14:43.520 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:14:43.698 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:14:43.698 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367),70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)[L],b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)]}]
00:14:43.699 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=10, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=268), Traces: [0ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [1ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [2ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [26ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [27ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [29ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [29ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [41ms] refreshing cache from master, [41ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [46ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [47ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [50ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [51ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [61ms] refreshing cache from master, [62ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [66ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [68ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [70ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [70ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [81ms] refreshing cache from master, [81ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [85ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [87ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [89ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [89ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [261ms] refreshing cache from master, [262ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [266ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@141882149(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@be0519d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@128719096 after retry RPC@370513551) -> (continuation of Deferred@1332949365 after retry RPC@1978169338) -> (continuation of Deferred@757248130 after retry RPC@457393531) -> (continuation of Deferred@644042545 after retry RPC@1832635577), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5d198624 -> wakeup thread Time-limited test -> (continuation of Deferred@128719096 after retry RPC@370513551) -> (continuation of Deferred@1332949365 after retry RPC@1978169338) -> (continuation of Deferred@757248130 after retry RPC@457393531) -> (continuation of Deferred@644042545 after retry RPC@1832635577))) after lookup
00:14:43.700 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] connecting to peer
00:14:43.701 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:14:43.701 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] cleaning up while in state CONNECTING due to: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:14:43.702 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:14:44.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:44.400851 30875 heartbeater.cc:499] Master 127.28.157.60:33393 was elected leader, sending a full tablet report...
00:14:44.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:44.628016 30895 raft_consensus.cc:493] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 70a127fea97b464a89d03c5848cf6637)
00:14:44.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:44.628507 30895 raft_consensus.cc:515] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:14:44.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:44.630404 30895 leader_election.cc:290] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b5d8d89741dd4568a4c7b210a4da72d2 (127.28.157.2:45951), 70a127fea97b464a89d03c5848cf6637 (127.28.157.1:37179)
00:14:44.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:44.638557 30285 leader_election.cc:336] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 70a127fea97b464a89d03c5848cf6637 (127.28.157.1:37179): Network error: Client connection negotiation failed: client connection to 127.28.157.1:37179: connect: Connection refused (error 111)
00:14:44.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:44.643453 30810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f4c1ce5aa4a4c8da912065e2decef6a" candidate_uuid: "094dbc44c6eb439083946b5ab0085ad0" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" is_pre_election: true
00:14:44.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:44.644196 30810 raft_consensus.cc:2468] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 094dbc44c6eb439083946b5ab0085ad0 in term 1.
00:14:44.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:44.645607 30289 leader_election.cc:304] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 094dbc44c6eb439083946b5ab0085ad0, b5d8d89741dd4568a4c7b210a4da72d2; no voters: 70a127fea97b464a89d03c5848cf6637
00:14:44.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:44.646292 30895 raft_consensus.cc:2804] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 2 FOLLOWER]: Leader pre-election won for term 3
00:14:44.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:44.646597 30895 raft_consensus.cc:493] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 70a127fea97b464a89d03c5848cf6637)
00:14:44.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:44.646829 30895 raft_consensus.cc:3060] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 2 FOLLOWER]: Advancing to term 3
00:14:44.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:44.651434 30895 raft_consensus.cc:515] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:14:44.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:44.653136 30895 leader_election.cc:290] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [CANDIDATE]: Term 3 election: Requested vote from peers b5d8d89741dd4568a4c7b210a4da72d2 (127.28.157.2:45951), 70a127fea97b464a89d03c5848cf6637 (127.28.157.1:37179)
00:14:44.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:44.654045 30810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f4c1ce5aa4a4c8da912065e2decef6a" candidate_uuid: "094dbc44c6eb439083946b5ab0085ad0" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "b5d8d89741dd4568a4c7b210a4da72d2"
00:14:44.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:44.654630 30810 raft_consensus.cc:3060] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [term 1 FOLLOWER]: Advancing to term 3
00:14:44.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:44.657621 30285 leader_election.cc:336] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 70a127fea97b464a89d03c5848cf6637 (127.28.157.1:37179): Network error: Client connection negotiation failed: client connection to 127.28.157.1:37179: connect: Connection refused (error 111)
00:14:44.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:44.666363 30810 raft_consensus.cc:2468] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 094dbc44c6eb439083946b5ab0085ad0 in term 3.
00:14:44.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:44.667779 30289 leader_election.cc:304] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 094dbc44c6eb439083946b5ab0085ad0, b5d8d89741dd4568a4c7b210a4da72d2; no voters: 70a127fea97b464a89d03c5848cf6637
00:14:44.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:44.668481 30895 raft_consensus.cc:2804] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 3 FOLLOWER]: Leader election won for term 3
00:14:44.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:44.669727 30895 raft_consensus.cc:697] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 3 LEADER]: Becoming Leader. State: Replica: 094dbc44c6eb439083946b5ab0085ad0, State: Running, Role: LEADER
00:14:44.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:44.671111 30895 consensus_queue.cc:237] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:14:44.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:44.680883 29608 catalog_manager.cc:5719] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 reported cstate change: term changed from 2 to 3, leader changed from 70a127fea97b464a89d03c5848cf6637 (127.28.157.1) to 094dbc44c6eb439083946b5ab0085ad0 (127.28.157.3). New cstate: current_term: 3 leader_uuid: "094dbc44c6eb439083946b5ab0085ad0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } health_report { overall_health: UNKNOWN } } }
00:14:44.977 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:14:44.977 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)[L],70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179),b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)]}]
00:14:44.978 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=12, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=1547), Traces: [0ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [1ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [2ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [26ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [27ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [29ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [29ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [41ms] refreshing cache from master, [41ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [46ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [47ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [50ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [51ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [61ms] refreshing cache from master, [62ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [66ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [68ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [70ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [70ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [81ms] refreshing cache from master, [81ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [85ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [87ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [89ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [89ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [261ms] refreshing cache from master, [262ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [266ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [268ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [271ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [271ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [1541ms] refreshing cache from master, [1541ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [1545ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@141882149(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@be0519d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@128719096 after retry RPC@370513551) -> (continuation of Deferred@1332949365 after retry RPC@1978169338) -> (continuation of Deferred@757248130 after retry RPC@457393531) -> (continuation of Deferred@644042545 after retry RPC@1832635577) -> (continuation of Deferred@535485992 after retry RPC@747083624), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5d198624 -> wakeup thread Time-limited test -> (continuation of Deferred@128719096 after retry RPC@370513551) -> (continuation of Deferred@1332949365 after retry RPC@1978169338) -> (continuation of Deferred@757248130 after retry RPC@457393531) -> (continuation of Deferred@644042545 after retry RPC@1832635577) -> (continuation of Deferred@535485992 after retry RPC@747083624))) after lookup
00:14:44.979 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)] connecting to peer
00:14:44.980 [DEBUG - kudu-nio-1] (Connection.java:757) [peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)] Successfully connected to peer
00:14:45.008 [DEBUG - kudu-nio-1] (Negotiator.java:918) Authenticated connection [id: 0xb27c448f, L:/127.0.0.1:34364 - R:/127.28.157.3:42367] using TOKEN/null
00:14:45.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:45.019416 30810 raft_consensus.cc:1275] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [term 3 FOLLOWER]: Refusing update from remote peer 094dbc44c6eb439083946b5ab0085ad0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
00:14:45.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:45.021039 30285 consensus_peers.cc:597] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 -> Peer 70a127fea97b464a89d03c5848cf6637 (127.28.157.1:37179): Couldn't send request to peer 70a127fea97b464a89d03c5848cf6637. Status: Network error: Client connection negotiation failed: client connection to 127.28.157.1:37179: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
00:14:45.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:45.021335 30901 consensus_queue.cc:1048] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 4, Time since last communication: 0.001s
00:14:45.124 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.28.157.1:37179
00:14:45.125 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

00:14:45.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:45.127417 29300 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu
00:14:45.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu
00:14:45.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/wal
00:14:45.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/data
00:14:45.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
00:14:45.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
00:14:45.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
00:14:45.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
00:14:45.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
00:14:45.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
00:14:45.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/logs
00:14:45.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/data/info.pb
00:14:45.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
00:14:45.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
00:14:45.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
00:14:45.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
00:14:45.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
00:14:45.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
00:14:45.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
00:14:45.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
00:14:45.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
00:14:45.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
00:14:45.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.28.157.1:37179
00:14:45.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.28.157.1
00:14:45.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.28.157.62:35965,127.28.157.61:44801,127.28.157.60:33393
00:14:45.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=35519
00:14:45.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.28.157.1
00:14:45.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.28.157.20:34671
00:14:45.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
00:14:45.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
00:14:45.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
00:14:45.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:45.503998 30937 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
00:14:45.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:45.504536 30937 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
00:14:45.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:45.505012 30937 flags.cc:432] Enabled unsafe flag: --never_fsync=true
00:14:45.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:45.540611 30937 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
00:14:45.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:45.541599 30937 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.157.1
00:14:45.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:45.580446 30937 tablet_server_runner.cc:78] Tablet server non-default flags:
00:14:45.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/logs
00:14:45.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
00:14:45.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
00:14:45.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
00:14:45.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.28.157.20:34671
00:14:45.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
00:14:45.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
00:14:45.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/data
00:14:45.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/wal
00:14:45.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
00:14:45.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
00:14:45.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.28.157.1:37179
00:14:45.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
00:14:45.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
00:14:45.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
00:14:45.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/data/info.pb
00:14:45.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.28.157.1
00:14:45.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=35519
00:14:45.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.28.157.62:35965,127.28.157.61:44801,127.28.157.60:33393
00:14:45.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
00:14:45.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
00:14:45.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
00:14:45.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
00:14:45.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
00:14:45.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.28.157.1
00:14:45.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
00:14:45.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
00:14:45.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
00:14:45.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision f7ed5b56c096f54e2f926bf94ceb7fc44c0c1329
00:14:45.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
00:14:45.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 11 Aug 2026 23:55:14 UTC on e7f111948823
00:14:45.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 17639
00:14:45.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
00:14:45.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:45.581857 30937 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
00:14:45.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:45.583865 30937 file_cache.cc:504] Constructed file cache file cache with capacity 419430
00:14:45.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:45.596827 30285 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.157.1:37179: connect: Connection refused (error 111) [suppressed 3 similar messages]
00:14:45.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:45.597452 30949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
00:14:46.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:45.598191 30953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
00:14:46.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:46.847910 30937 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.249s	user 0.001s	sys 0.007s
00:14:46.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:46.848130 30958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
00:14:46.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:46.848800 30937 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.250s	user 0.002s	sys 0.007s
00:14:46.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:46.849098 30937 server_base.cc:1061] running on GCE node
00:14:46.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:46.850320 30937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
00:14:46.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:46.852943 30937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
00:14:46.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:46.854699 30937 hybrid_clock.cc:648] HybridClock initialized: now 1786493686854591 us; error 89 us; skew 500 ppm
00:14:46.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:46.855822 30937 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
00:14:46.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:46.866894 30937 webserver.cc:533] Webserver started at http://127.28.157.1:35519/ using document root <none> and password file <none>
00:14:46.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:46.868247 30937 fs_manager.cc:362] Metadata directory not provided
00:14:46.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:46.868652 30937 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
00:14:46.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:46.878502 30937 fs_manager.cc:714] Time spent opening directory manager: real 0.006s	user 0.006s	sys 0.001s
00:14:46.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:46.884445 30973 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
00:14:46.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:46.885977 30937 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.005s	sys 0.002s
00:14:46.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:46.886359 30937 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/data,/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/wal
00:14:46.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "70a127fea97b464a89d03c5848cf6637"
00:14:46.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 00:14:32 on dist-test-slave-t7g5"
00:14:46.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:46.888443 30937 fs_report.cc:389] FS layout report
00:14:46.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
00:14:46.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/wal
00:14:46.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/wal
00:14:46.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/data/data
00:14:46.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
00:14:46.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
00:14:46.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
00:14:46.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
00:14:46.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
00:14:46.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
00:14:46.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
00:14:46.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
00:14:46.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
00:14:46.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
00:14:46.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
00:14:46.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:46.973068 30937 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
00:14:46.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:46.974869 30937 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
00:14:46.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:46.975431 30937 kserver.cc:163] Server-wide thread pool size limit: 3276
00:14:46.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:46.978260 30937 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
00:14:46.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:46.984678 30992 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
00:14:46.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:46.994324 30937 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
00:14:46.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:46.994575 30937 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.012s	user 0.001s	sys 0.001s
00:14:46.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:46.994815 30937 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
00:14:47.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:47.000139 30937 ts_tablet_manager.cc:616] Registered 1 tablets
00:14:47.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:47.000331 30937 ts_tablet_manager.cc:595] Time spent register tablets: real 0.006s	user 0.005s	sys 0.000s
00:14:47.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:47.000865 30992 tablet_bootstrap.cc:492] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637: Bootstrap starting.
00:14:47.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:47.063015 30992 log.cc:826] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637: Log is configured to *not* fsync() on all Append() calls
00:14:47.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:47.112673 30992 tablet_bootstrap.cc:492] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=6 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
00:14:47.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:47.113902 30992 tablet_bootstrap.cc:492] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637: Bootstrap complete.
00:14:47.116 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:47.115984 30992 ts_tablet_manager.cc:1403] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637: Time spent bootstrapping tablet: real 0.116s	user 0.064s	sys 0.044s
00:14:47.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:47.141578 30992 raft_consensus.cc:359] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:14:47.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:47.144656 30992 raft_consensus.cc:740] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70a127fea97b464a89d03c5848cf6637, State: Initialized, Role: FOLLOWER
00:14:47.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:47.145761 30992 consensus_queue.cc:260] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:14:47.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:47.151237 30992 ts_tablet_manager.cc:1434] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637: Time spent starting tablet: real 0.035s	user 0.033s	sys 0.000s
00:14:47.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:47.183499 30937 rpc_server.cc:307] RPC server started. Bound to: 127.28.157.1:37179
00:14:47.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:47.184056 31180 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.157.1:37179 every 8 connection(s)
00:14:47.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:47.186136 30937 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/data/info.pb
00:14:47.190 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
00:14:47.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:47.188377 29300 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu as pid 30937
00:14:47.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:47.229770 31183 heartbeater.cc:344] Connected to a master server at 127.28.157.60:33393
00:14:47.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:47.230768 31183 heartbeater.cc:461] Registering TS with master...
00:14:47.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:47.232596 31183 heartbeater.cc:507] Master 127.28.157.60:33393 requested a full tablet report, sending...
00:14:47.234 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 9 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
00:14:47.235 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.28.157.3:42367
00:14:47.236 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

00:14:47.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:47.236490 29300 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu with pid 30256
00:14:47.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:47.278777 31096 raft_consensus.cc:3060] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [term 2 FOLLOWER]: Advancing to term 3
00:14:47.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:47.280337 29608 ts_manager.cc:194] Re-registered known tserver with Master: 70a127fea97b464a89d03c5848cf6637 (127.28.157.1:37179)
00:14:47.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:47.285162 29608 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.28.157.1:60179
00:14:47.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:47.286657 31184 heartbeater.cc:344] Connected to a master server at 127.28.157.61:44801
00:14:47.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:47.287071 31184 heartbeater.cc:461] Registering TS with master...
00:14:47.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:47.288113 31184 heartbeater.cc:507] Master 127.28.157.61:44801 requested a full tablet report, sending...
00:14:47.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:47.292161 31183 heartbeater.cc:499] Master 127.28.157.60:33393 was elected leader, sending a full tablet report...
00:14:47.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:47.296845 31197 heartbeater.cc:344] Connected to a master server at 127.28.157.62:35965
00:14:47.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:47.297276 31197 heartbeater.cc:461] Registering TS with master...
00:14:47.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:47.298151 31197 heartbeater.cc:507] Master 127.28.157.62:35965 requested a full tablet report, sending...
00:14:47.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:47.305490 31206 mvcc.cc:204] Tried to move back new op lower bound from 7317478133821263872 to 7317478132420947968. Current Snapshot: MvccSnapshot[applied={T|T < 7317478133821263872}]
00:14:47.313 [DEBUG - kudu-nio-1] (Connection.java:239) [peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)] handling channelInactive
00:14:47.314 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)] cleaning up while in state READY due to: connection closed
00:14:47.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:47.315433 30978 connection.cc:466] server connection from 127.28.157.3:41795 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.28.157.3:41795 (request call id 0) could send its response
00:14:47.317 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
00:14:47.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:47.320919 29360 ts_manager.cc:194] Re-registered known tserver with Master: 70a127fea97b464a89d03c5848cf6637 (127.28.157.1:37179)
00:14:47.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:47.321522 29481 ts_manager.cc:194] Re-registered known tserver with Master: 70a127fea97b464a89d03c5848cf6637 (127.28.157.1:37179)
00:14:47.322 [DEBUG - Time-limited test] (Connection.java:748) [peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)] connecting to peer
00:14:47.323 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367
00:14:47.324 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)] cleaning up while in state CONNECTING due to: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367
00:14:47.324 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367
00:14:47.338 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:14:47.338 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)[L],70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179),b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)]}]
00:14:47.339 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=2, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=21), Traces: [0ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [6ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [6ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [14ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [20ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@11201398(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@1536d354 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5d95bdf5 -> wakeup thread Time-limited test)) after lookup
00:14:47.340 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)] connecting to peer
00:14:47.341 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367
00:14:47.341 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)] cleaning up while in state CONNECTING due to: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367
00:14:47.341 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367
00:14:47.357 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:14:47.357 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)[L],70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179),b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)]}]
00:14:47.357 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=4, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=40), Traces: [0ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [6ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [6ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [14ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [20ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [21ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [23ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [23ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [34ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [38ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@11201398(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@1536d354 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@374200658 after retry RPC@1724363899), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5d95bdf5 -> wakeup thread Time-limited test -> (continuation of Deferred@374200658 after retry RPC@1724363899))) after lookup
00:14:47.358 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)] connecting to peer
00:14:47.359 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367
00:14:47.359 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)] cleaning up while in state CONNECTING due to: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367
00:14:47.359 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367
00:14:47.397 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:14:47.398 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)[L],70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179),b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)]}]
00:14:47.398 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=6, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=80), Traces: [0ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [6ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [6ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [14ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [20ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [21ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [23ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [23ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [34ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [38ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [40ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [41ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [41ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [74ms] refreshing cache from master, [74ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [79ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@11201398(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@1536d354 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@374200658 after retry RPC@1724363899) -> (continuation of Deferred@2093963723 after retry RPC@661671493), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5d95bdf5 -> wakeup thread Time-limited test -> (continuation of Deferred@374200658 after retry RPC@1724363899) -> (continuation of Deferred@2093963723 after retry RPC@661671493))) after lookup
00:14:47.399 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)] connecting to peer
00:14:47.400 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367
00:14:47.400 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)] cleaning up while in state CONNECTING due to: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367
00:14:47.400 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367
00:14:47.498 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:14:47.498 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)[L],70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179),b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)]}]
00:14:47.499 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=8, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=181), Traces: [0ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [6ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [6ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [14ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [20ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [21ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [23ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [23ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [34ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [38ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [40ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [41ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [41ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [74ms] refreshing cache from master, [74ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [79ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [80ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [82ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [82ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [174ms] refreshing cache from master, [174ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [179ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@11201398(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@1536d354 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@374200658 after retry RPC@1724363899) -> (continuation of Deferred@2093963723 after retry RPC@661671493) -> (continuation of Deferred@167391167 after retry RPC@744315555), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5d95bdf5 -> wakeup thread Time-limited test -> (continuation of Deferred@374200658 after retry RPC@1724363899) -> (continuation of Deferred@2093963723 after retry RPC@661671493) -> (continuation of Deferred@167391167 after retry RPC@744315555))) after lookup
00:14:47.499 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)] connecting to peer
00:14:47.500 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367
00:14:47.500 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)] cleaning up while in state CONNECTING due to: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367
00:14:47.501 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367
00:14:47.917 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:14:47.919 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)[L],70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179),b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)]}]
00:14:47.920 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=10, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=602), Traces: [0ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [6ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [6ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [14ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [20ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [21ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [23ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [23ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [34ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [38ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [40ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [41ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [41ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [74ms] refreshing cache from master, [74ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [79ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [80ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [82ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [82ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [174ms] refreshing cache from master, [174ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [179ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [181ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [183ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [183ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [594ms] refreshing cache from master, [594ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [598ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@11201398(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@1536d354 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@374200658 after retry RPC@1724363899) -> (continuation of Deferred@2093963723 after retry RPC@661671493) -> (continuation of Deferred@167391167 after retry RPC@744315555) -> (continuation of Deferred@624802134 after retry RPC@1668511504), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5d95bdf5 -> wakeup thread Time-limited test -> (continuation of Deferred@374200658 after retry RPC@1724363899) -> (continuation of Deferred@2093963723 after retry RPC@661671493) -> (continuation of Deferred@167391167 after retry RPC@744315555) -> (continuation of Deferred@624802134 after retry RPC@1668511504))) after lookup
00:14:47.921 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)] connecting to peer
00:14:47.922 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367
00:14:47.923 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)] cleaning up while in state CONNECTING due to: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367
00:14:47.923 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367
00:14:48.198 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:14:48.199 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)[L],70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179),b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)]}]
00:14:48.199 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=12, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=881), Traces: [0ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [6ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [6ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [14ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [20ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [21ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [23ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [23ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [34ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [38ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [40ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [41ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [41ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [74ms] refreshing cache from master, [74ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [79ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [80ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [82ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [82ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [174ms] refreshing cache from master, [174ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [179ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [181ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [183ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [183ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [594ms] refreshing cache from master, [594ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [598ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [602ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [605ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [605ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [874ms] refreshing cache from master, [874ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [880ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@11201398(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@1536d354 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@374200658 after retry RPC@1724363899) -> (continuation of Deferred@2093963723 after retry RPC@661671493) -> (continuation of Deferred@167391167 after retry RPC@744315555) -> (continuation of Deferred@624802134 after retry RPC@1668511504) -> (continuation of Deferred@840604607 after retry RPC@1252634907), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5d95bdf5 -> wakeup thread Time-limited test -> (continuation of Deferred@374200658 after retry RPC@1724363899) -> (continuation of Deferred@2093963723 after retry RPC@661671493) -> (continuation of Deferred@167391167 after retry RPC@744315555) -> (continuation of Deferred@624802134 after retry RPC@1668511504) -> (continuation of Deferred@840604607 after retry RPC@1252634907))) after lookup
00:14:48.200 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)] connecting to peer
00:14:48.202 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367
00:14:48.202 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)] cleaning up while in state CONNECTING due to: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367
00:14:48.202 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367
00:14:48.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:48.597790 31214 raft_consensus.cc:493] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 094dbc44c6eb439083946b5ab0085ad0)
00:14:48.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:48.598209 31214 raft_consensus.cc:515] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:14:48.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:48.600723 31214 leader_election.cc:290] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 094dbc44c6eb439083946b5ab0085ad0 (127.28.157.3:42367), 70a127fea97b464a89d03c5848cf6637 (127.28.157.1:37179)
00:14:48.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:48.604923 30686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.157.3:42367: connect: Connection refused (error 111)
00:14:48.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:48.609263 30686 leader_election.cc:336] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 094dbc44c6eb439083946b5ab0085ad0 (127.28.157.3:42367): Network error: Client connection negotiation failed: client connection to 127.28.157.3:42367: connect: Connection refused (error 111)
00:14:48.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:48.613394 31096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f4c1ce5aa4a4c8da912065e2decef6a" candidate_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "70a127fea97b464a89d03c5848cf6637" is_pre_election: true
00:14:48.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:48.615293 30684 leader_election.cc:304] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b5d8d89741dd4568a4c7b210a4da72d2; no voters: 094dbc44c6eb439083946b5ab0085ad0, 70a127fea97b464a89d03c5848cf6637
00:14:48.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:48.616020 31214 raft_consensus.cc:2749] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
00:14:48.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:48.794970 31222 raft_consensus.cc:493] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 094dbc44c6eb439083946b5ab0085ad0)
00:14:48.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:48.795511 31222 raft_consensus.cc:515] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:14:48.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:48.798058 31222 leader_election.cc:290] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b5d8d89741dd4568a4c7b210a4da72d2 (127.28.157.2:45951), 094dbc44c6eb439083946b5ab0085ad0 (127.28.157.3:42367)
00:14:48.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:48.802202 30978 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.157.3:42367: connect: Connection refused (error 111)
00:14:48.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:48.807047 30978 leader_election.cc:336] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 094dbc44c6eb439083946b5ab0085ad0 (127.28.157.3:42367): Network error: Client connection negotiation failed: client connection to 127.28.157.3:42367: connect: Connection refused (error 111)
00:14:48.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:48.810155 30810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f4c1ce5aa4a4c8da912065e2decef6a" candidate_uuid: "70a127fea97b464a89d03c5848cf6637" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" is_pre_election: true
00:14:48.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:48.810812 30810 raft_consensus.cc:2468] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70a127fea97b464a89d03c5848cf6637 in term 3.
00:14:48.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:48.812064 30980 leader_election.cc:304] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 70a127fea97b464a89d03c5848cf6637, b5d8d89741dd4568a4c7b210a4da72d2; no voters: 094dbc44c6eb439083946b5ab0085ad0
00:14:48.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:48.813021 31222 raft_consensus.cc:2804] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [term 3 FOLLOWER]: Leader pre-election won for term 4
00:14:48.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:48.813448 31222 raft_consensus.cc:493] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 094dbc44c6eb439083946b5ab0085ad0)
00:14:48.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:48.813779 31222 raft_consensus.cc:3060] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [term 3 FOLLOWER]: Advancing to term 4
00:14:48.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:48.819638 31222 raft_consensus.cc:515] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:14:48.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:48.821269 31222 leader_election.cc:290] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [CANDIDATE]: Term 4 election: Requested vote from peers b5d8d89741dd4568a4c7b210a4da72d2 (127.28.157.2:45951), 094dbc44c6eb439083946b5ab0085ad0 (127.28.157.3:42367)
00:14:48.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:48.822134 30810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f4c1ce5aa4a4c8da912065e2decef6a" candidate_uuid: "70a127fea97b464a89d03c5848cf6637" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "b5d8d89741dd4568a4c7b210a4da72d2"
00:14:48.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:48.822624 30810 raft_consensus.cc:3060] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [term 3 FOLLOWER]: Advancing to term 4
00:14:48.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:48.825157 30978 leader_election.cc:336] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 094dbc44c6eb439083946b5ab0085ad0 (127.28.157.3:42367): Network error: Client connection negotiation failed: client connection to 127.28.157.3:42367: connect: Connection refused (error 111)
00:14:48.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:48.828259 30810 raft_consensus.cc:2468] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70a127fea97b464a89d03c5848cf6637 in term 4.
00:14:48.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:48.829331 30980 leader_election.cc:304] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 70a127fea97b464a89d03c5848cf6637, b5d8d89741dd4568a4c7b210a4da72d2; no voters: 094dbc44c6eb439083946b5ab0085ad0
00:14:48.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:48.829993 31222 raft_consensus.cc:2804] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [term 4 FOLLOWER]: Leader election won for term 4
00:14:48.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:48.831550 31222 raft_consensus.cc:697] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [term 4 LEADER]: Becoming Leader. State: Replica: 70a127fea97b464a89d03c5848cf6637, State: Running, Role: LEADER
00:14:48.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:48.832456 31222 consensus_queue.cc:237] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 12, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:14:48.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:48.842015 29608 catalog_manager.cc:5719] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 reported cstate change: term changed from 3 to 4, leader changed from 094dbc44c6eb439083946b5ab0085ad0 (127.28.157.3) to 70a127fea97b464a89d03c5848cf6637 (127.28.157.1). New cstate: current_term: 4 leader_uuid: "70a127fea97b464a89d03c5848cf6637" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } health_report { overall_health: HEALTHY } } }
00:14:49.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:49.355305 30978 consensus_peers.cc:597] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 -> Peer 094dbc44c6eb439083946b5ab0085ad0 (127.28.157.3:42367): Couldn't send request to peer 094dbc44c6eb439083946b5ab0085ad0. Status: Network error: Client connection negotiation failed: client connection to 127.28.157.3:42367: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
00:14:49.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:49.359154 31177 debug-util.cc:398] Leaking SignalData structure 0x7b08000bfce0 after lost signal to thread 30939
00:14:49.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:49.425997 30810 raft_consensus.cc:1275] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [term 4 FOLLOWER]: Refusing update from remote peer 70a127fea97b464a89d03c5848cf6637: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 4 index: 13. (index mismatch)
00:14:49.428 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:49.428086 31247 consensus_queue.cc:1048] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
00:14:49.618 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:14:49.618 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367),70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)[L],b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)]}]
00:14:49.618 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=14, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=2300), Traces: [0ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [6ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [6ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [14ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [20ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [21ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [23ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [23ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [34ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [38ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [40ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [41ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [41ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [74ms] refreshing cache from master, [74ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [79ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [80ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [82ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [82ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [174ms] refreshing cache from master, [174ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [179ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [181ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [183ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [183ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [594ms] refreshing cache from master, [594ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [598ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [602ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [605ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [605ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [874ms] refreshing cache from master, [874ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [880ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [882ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [884ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [884ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [2295ms] refreshing cache from master, [2295ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [2299ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@11201398(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@1536d354 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@374200658 after retry RPC@1724363899) -> (continuation of Deferred@2093963723 after retry RPC@661671493) -> (continuation of Deferred@167391167 after retry RPC@744315555) -> (continuation of Deferred@624802134 after retry RPC@1668511504) -> (continuation of Deferred@840604607 after retry RPC@1252634907) -> (continuation of Deferred@657241035 after retry RPC@545980252), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5d95bdf5 -> wakeup thread Time-limited test -> (continuation of Deferred@374200658 after retry RPC@1724363899) -> (continuation of Deferred@2093963723 after retry RPC@661671493) -> (continuation of Deferred@167391167 after retry RPC@744315555) -> (continuation of Deferred@624802134 after retry RPC@1668511504) -> (continuation of Deferred@840604607 after retry RPC@1252634907) -> (continuation of Deferred@657241035 after retry RPC@545980252))) after lookup
00:14:49.619 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] connecting to peer
00:14:49.620 [DEBUG - kudu-nio-1] (Connection.java:757) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] Successfully connected to peer
00:14:49.666 [DEBUG - kudu-nio-1] (Negotiator.java:918) Authenticated connection [id: 0x7e0107ca, L:/127.0.0.1:35912 - R:/127.28.157.1:37179] using TOKEN/null
00:14:49.753 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.28.157.3:42367
00:14:49.753 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

00:14:49.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:49.755188 29300 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu
00:14:49.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu
00:14:49.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-2/wal
00:14:49.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-2/data
00:14:49.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
00:14:49.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
00:14:49.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
00:14:49.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
00:14:49.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
00:14:49.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
00:14:49.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-2/logs
00:14:49.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-2/data/info.pb
00:14:49.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
00:14:49.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
00:14:49.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
00:14:49.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
00:14:49.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
00:14:49.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
00:14:49.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
00:14:49.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
00:14:49.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
00:14:49.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
00:14:49.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.28.157.3:42367
00:14:49.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.28.157.3
00:14:49.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.28.157.62:35965,127.28.157.61:44801,127.28.157.60:33393
00:14:49.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=36411
00:14:49.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.28.157.3
00:14:49.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.28.157.20:34671
00:14:49.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
00:14:49.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
00:14:49.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
00:14:50.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:50.127673 31266 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
00:14:50.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:50.128221 31266 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
00:14:50.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:50.128762 31266 flags.cc:432] Enabled unsafe flag: --never_fsync=true
00:14:50.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:50.165802 31266 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
00:14:50.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:50.166741 31266 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.157.3
00:14:50.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:50.211675 31266 tablet_server_runner.cc:78] Tablet server non-default flags:
00:14:50.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-2/logs
00:14:50.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
00:14:50.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
00:14:50.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
00:14:50.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.28.157.20:34671
00:14:50.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
00:14:50.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
00:14:50.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-2/data
00:14:50.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-2/wal
00:14:50.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
00:14:50.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
00:14:50.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.28.157.3:42367
00:14:50.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
00:14:50.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
00:14:50.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
00:14:50.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-2/data/info.pb
00:14:50.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.28.157.3
00:14:50.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=36411
00:14:50.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.28.157.62:35965,127.28.157.61:44801,127.28.157.60:33393
00:14:50.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
00:14:50.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
00:14:50.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
00:14:50.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
00:14:50.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
00:14:50.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.28.157.3
00:14:50.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
00:14:50.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
00:14:50.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
00:14:50.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision f7ed5b56c096f54e2f926bf94ceb7fc44c0c1329
00:14:50.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
00:14:50.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 11 Aug 2026 23:55:14 UTC on e7f111948823
00:14:50.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 17639
00:14:50.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
00:14:50.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:50.213470 31266 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
00:14:50.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:50.215225 31266 file_cache.cc:504] Constructed file cache file cache with capacity 419430
00:14:50.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:50.232594 31282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
00:14:51.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:51.952945 30978 consensus_peers.cc:597] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 -> Peer 094dbc44c6eb439083946b5ab0085ad0 (127.28.157.3:42367): Couldn't send request to peer 094dbc44c6eb439083946b5ab0085ad0. Status: Network error: Client connection negotiation failed: client connection to 127.28.157.3:42367: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
00:14:52.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:52.057088 30870 debug-util.cc:398] Leaking SignalData structure 0x7b08000bfc00 after lost signal to thread 30656
00:14:52.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:52.058581 30870 debug-util.cc:398] Leaking SignalData structure 0x7b08000bf3a0 after lost signal to thread 30874
00:14:52.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:51.633337 31281 debug-util.cc:398] Leaking SignalData structure 0x7b080002ede0 after lost signal to thread 31266
00:14:52.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:52.191659 31266 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.957s	user 0.763s	sys 1.110s
00:14:52.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:52.194082 31266 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.960s	user 0.764s	sys 1.110s
00:14:52.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:52.198037 31284 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1960 milliseconds
00:14:52.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:52.198498 31285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
00:14:52.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:52.199882 31266 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
00:14:52.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:50.234512 31283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
00:14:52.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:52.204995 31266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
00:14:52.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:52.208040 31266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
00:14:52.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:52.209534 31266 hybrid_clock.cc:648] HybridClock initialized: now 1786493692209369 us; error 170 us; skew 500 ppm
00:14:52.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:52.210701 31266 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
00:14:52.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:52.219223 31266 webserver.cc:533] Webserver started at http://127.28.157.3:36411/ using document root <none> and password file <none>
00:14:52.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:52.220494 31266 fs_manager.cc:362] Metadata directory not provided
00:14:52.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:52.220789 31266 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
00:14:52.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:52.232290 31266 fs_manager.cc:714] Time spent opening directory manager: real 0.007s	user 0.006s	sys 0.003s
00:14:52.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:52.238972 31305 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
00:14:52.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:52.240595 31266 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.001s	sys 0.003s
00:14:52.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:52.241024 31266 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-2/data,/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-2/wal
00:14:52.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "094dbc44c6eb439083946b5ab0085ad0"
00:14:52.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 00:14:36 on dist-test-slave-t7g5"
00:14:52.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:52.243858 31266 fs_report.cc:389] FS layout report
00:14:52.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
00:14:52.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-2/wal
00:14:52.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-2/wal
00:14:52.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-2/data/data
00:14:52.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
00:14:52.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
00:14:52.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
00:14:52.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
00:14:52.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
00:14:52.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
00:14:52.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
00:14:52.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
00:14:52.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
00:14:52.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
00:14:52.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
00:14:52.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:52.333642 31266 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
00:14:52.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:52.335727 31266 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
00:14:52.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:52.336352 31266 kserver.cc:163] Server-wide thread pool size limit: 3276
00:14:52.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:52.340095 31266 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
00:14:52.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:52.346876 31321 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
00:14:52.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:52.359267 31266 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
00:14:52.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:52.359666 31266 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.015s	user 0.000s	sys 0.002s
00:14:52.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:52.360070 31266 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
00:14:52.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:52.365805 31266 ts_tablet_manager.cc:616] Registered 1 tablets
00:14:52.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:52.366047 31266 ts_tablet_manager.cc:595] Time spent register tablets: real 0.006s	user 0.007s	sys 0.000s
00:14:52.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:52.366572 31321 tablet_bootstrap.cc:492] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0: Bootstrap starting.
00:14:52.446 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:52.445583 31321 log.cc:826] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0: Log is configured to *not* fsync() on all Append() calls
00:14:52.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:52.512566 31321 tablet_bootstrap.cc:492] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=9 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
00:14:52.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:52.513687 31321 tablet_bootstrap.cc:492] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0: Bootstrap complete.
00:14:52.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:52.515420 31321 ts_tablet_manager.cc:1403] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0: Time spent bootstrapping tablet: real 0.149s	user 0.107s	sys 0.029s
00:14:52.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:52.540189 31321 raft_consensus.cc:359] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:14:52.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:52.543671 31321 raft_consensus.cc:740] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 094dbc44c6eb439083946b5ab0085ad0, State: Initialized, Role: FOLLOWER
00:14:52.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:52.545456 31321 consensus_queue.cc:260] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 3.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:14:52.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:52.549069 31321 ts_tablet_manager.cc:1434] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0: Time spent starting tablet: real 0.033s	user 0.029s	sys 0.003s
00:14:52.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:52.613165 31266 rpc_server.cc:307] RPC server started. Bound to: 127.28.157.3:42367
00:14:52.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:52.613246 31514 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.157.3:42367 every 8 connection(s)
00:14:52.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:52.617142 31266 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-2/data/info.pb
00:14:52.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:52.624454 29300 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu as pid 31266
00:14:52.625 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
00:14:52.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:52.670878 31516 heartbeater.cc:344] Connected to a master server at 127.28.157.60:33393
00:14:52.671 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 12 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
00:14:52.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:52.671402 31516 heartbeater.cc:461] Registering TS with master...
00:14:52.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:52.672930 31516 heartbeater.cc:507] Master 127.28.157.60:33393 requested a full tablet report, sending...
00:14:52.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:52.677320 29607 ts_manager.cc:194] Re-registered known tserver with Master: 094dbc44c6eb439083946b5ab0085ad0 (127.28.157.3:42367)
00:14:52.698 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.28.157.1:37179
00:14:52.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:52.680495 29607 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.28.157.3:37235
00:14:52.698 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

00:14:52.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:52.683136 31519 heartbeater.cc:344] Connected to a master server at 127.28.157.62:35965
00:14:52.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:52.683516 31519 heartbeater.cc:461] Registering TS with master...
00:14:52.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:52.684438 31519 heartbeater.cc:507] Master 127.28.157.62:35965 requested a full tablet report, sending...
00:14:52.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:52.689818 29360 ts_manager.cc:194] Re-registered known tserver with Master: 094dbc44c6eb439083946b5ab0085ad0 (127.28.157.3:42367)
00:14:52.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:52.699375 29300 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu with pid 30937
00:14:52.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:52.701938 31517 heartbeater.cc:344] Connected to a master server at 127.28.157.61:44801
00:14:52.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:52.702261 31517 heartbeater.cc:461] Registering TS with master...
00:14:52.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:52.702813 31517 heartbeater.cc:507] Master 127.28.157.61:44801 requested a full tablet report, sending...
00:14:52.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:52.705902 29481 ts_manager.cc:194] Re-registered known tserver with Master: 094dbc44c6eb439083946b5ab0085ad0 (127.28.157.3:42367)
00:14:52.806 [DEBUG - kudu-nio-1] (Connection.java:239) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] handling channelInactive
00:14:52.806 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
00:14:52.807 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] cleaning up while in state READY due to: connection closed
00:14:52.807 [DEBUG - Time-limited test] (Connection.java:748) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] connecting to peer
00:14:52.809 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:14:52.809 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] cleaning up while in state CONNECTING due to: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:14:52.809 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:14:52.817 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:14:52.818 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367),70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)[L],b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)]}]
00:14:52.818 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=2, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=11), Traces: [0ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [2ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [2ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [10ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@1359238655(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@ed4ef8b -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1105e955 -> wakeup thread Time-limited test)) after lookup
00:14:52.819 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] connecting to peer
00:14:52.820 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:14:52.820 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] cleaning up while in state CONNECTING due to: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:14:52.820 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:14:52.837 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:14:52.838 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367),70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)[L],b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)]}]
00:14:52.838 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=4, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=31), Traces: [0ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [2ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [2ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [10ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [11ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [13ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [13ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [30ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@1359238655(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@ed4ef8b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@411442383 after retry RPC@653722059), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1105e955 -> wakeup thread Time-limited test -> (continuation of Deferred@411442383 after retry RPC@653722059))) after lookup
00:14:52.839 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] connecting to peer
00:14:52.842 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:14:52.842 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] cleaning up while in state CONNECTING due to: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:14:52.842 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:14:52.862 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:14:52.862 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367),70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)[L],b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)]}]
00:14:52.863 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=6, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=55), Traces: [0ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [2ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [2ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [10ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [11ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [13ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [13ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [30ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [32ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [35ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [35ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [45ms] refreshing cache from master, [45ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [55ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@1359238655(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@ed4ef8b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@411442383 after retry RPC@653722059) -> (continuation of Deferred@727009250 after retry RPC@823771659), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1105e955 -> wakeup thread Time-limited test -> (continuation of Deferred@411442383 after retry RPC@653722059) -> (continuation of Deferred@727009250 after retry RPC@823771659))) after lookup
00:14:52.863 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] connecting to peer
00:14:52.866 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:14:52.866 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] cleaning up while in state CONNECTING due to: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:14:52.866 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:14:52.999 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:14:52.999 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367),70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)[L],b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)]}]
00:14:53.001 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=8, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=193), Traces: [0ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [2ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [2ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [10ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [11ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [13ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [13ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [30ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [32ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [35ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [35ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [45ms] refreshing cache from master, [45ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [55ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [56ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [59ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [59ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [186ms] refreshing cache from master, [186ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [191ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@1359238655(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@ed4ef8b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@411442383 after retry RPC@653722059) -> (continuation of Deferred@727009250 after retry RPC@823771659) -> (continuation of Deferred@407645932 after retry RPC@1731898996), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1105e955 -> wakeup thread Time-limited test -> (continuation of Deferred@411442383 after retry RPC@653722059) -> (continuation of Deferred@727009250 after retry RPC@823771659) -> (continuation of Deferred@407645932 after retry RPC@1731898996))) after lookup
00:14:53.001 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] connecting to peer
00:14:53.003 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:14:53.003 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] cleaning up while in state CONNECTING due to: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:14:53.003 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:14:53.457 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:14:53.457 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367),70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)[L],b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)]}]
00:14:53.458 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=10, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=650), Traces: [0ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [2ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [2ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [10ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [11ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [13ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [13ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [30ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [32ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [35ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [35ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [45ms] refreshing cache from master, [45ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [55ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [56ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [59ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [59ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [186ms] refreshing cache from master, [186ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [191ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [194ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [196ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [196ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [645ms] refreshing cache from master, [646ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [650ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@1359238655(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@ed4ef8b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@411442383 after retry RPC@653722059) -> (continuation of Deferred@727009250 after retry RPC@823771659) -> (continuation of Deferred@407645932 after retry RPC@1731898996) -> (continuation of Deferred@1937049283 after retry RPC@1091456983), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1105e955 -> wakeup thread Time-limited test -> (continuation of Deferred@411442383 after retry RPC@653722059) -> (continuation of Deferred@727009250 after retry RPC@823771659) -> (continuation of Deferred@407645932 after retry RPC@1731898996) -> (continuation of Deferred@1937049283 after retry RPC@1091456983))) after lookup
00:14:53.458 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] connecting to peer
00:14:53.459 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:14:53.459 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] cleaning up while in state CONNECTING due to: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:14:53.460 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:14:53.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:53.697468 31516 heartbeater.cc:499] Master 127.28.157.60:33393 was elected leader, sending a full tablet report...
00:14:53.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:53.811808 31542 raft_consensus.cc:493] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 70a127fea97b464a89d03c5848cf6637)
00:14:53.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:53.812525 31542 raft_consensus.cc:515] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:14:53.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:53.815407 31542 leader_election.cc:290] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 094dbc44c6eb439083946b5ab0085ad0 (127.28.157.3:42367), 70a127fea97b464a89d03c5848cf6637 (127.28.157.1:37179)
00:14:53.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:53.820875 30684 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.157.1:37179: connect: Connection refused (error 111)
00:14:53.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:53.825107 30684 leader_election.cc:336] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 70a127fea97b464a89d03c5848cf6637 (127.28.157.1:37179): Network error: Client connection negotiation failed: client connection to 127.28.157.1:37179: connect: Connection refused (error 111)
00:14:53.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:53.832111 31433 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f4c1ce5aa4a4c8da912065e2decef6a" candidate_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "094dbc44c6eb439083946b5ab0085ad0" is_pre_election: true
00:14:53.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:53.833051 31433 raft_consensus.cc:2468] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5d8d89741dd4568a4c7b210a4da72d2 in term 3.
00:14:53.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:53.834836 30686 leader_election.cc:304] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 094dbc44c6eb439083946b5ab0085ad0, b5d8d89741dd4568a4c7b210a4da72d2; no voters: 70a127fea97b464a89d03c5848cf6637
00:14:53.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:53.835981 31542 raft_consensus.cc:2804] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [term 4 FOLLOWER]: Leader pre-election won for term 5
00:14:53.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:53.836309 31542 raft_consensus.cc:493] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 70a127fea97b464a89d03c5848cf6637)
00:14:53.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:53.836643 31542 raft_consensus.cc:3060] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [term 4 FOLLOWER]: Advancing to term 5
00:14:53.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:53.844164 31542 raft_consensus.cc:515] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:14:53.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:53.846266 31542 leader_election.cc:290] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [CANDIDATE]: Term 5 election: Requested vote from peers 094dbc44c6eb439083946b5ab0085ad0 (127.28.157.3:42367), 70a127fea97b464a89d03c5848cf6637 (127.28.157.1:37179)
00:14:53.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:53.850126 31433 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f4c1ce5aa4a4c8da912065e2decef6a" candidate_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "094dbc44c6eb439083946b5ab0085ad0"
00:14:53.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:53.850732 31433 raft_consensus.cc:3060] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 3 FOLLOWER]: Advancing to term 5
00:14:53.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:53.852298 30684 leader_election.cc:336] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 70a127fea97b464a89d03c5848cf6637 (127.28.157.1:37179): Network error: Client connection negotiation failed: client connection to 127.28.157.1:37179: connect: Connection refused (error 111)
00:14:53.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:53.861186 31433 raft_consensus.cc:2468] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5d8d89741dd4568a4c7b210a4da72d2 in term 5.
00:14:53.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:53.862846 30686 leader_election.cc:304] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 094dbc44c6eb439083946b5ab0085ad0, b5d8d89741dd4568a4c7b210a4da72d2; no voters: 70a127fea97b464a89d03c5848cf6637
00:14:53.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:53.863842 31542 raft_consensus.cc:2804] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [term 5 FOLLOWER]: Leader election won for term 5
00:14:53.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:53.865736 31542 raft_consensus.cc:697] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [term 5 LEADER]: Becoming Leader. State: Replica: b5d8d89741dd4568a4c7b210a4da72d2, State: Running, Role: LEADER
00:14:53.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:53.867513 31542 consensus_queue.cc:237] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 4.16, Last appended by leader: 16, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:14:53.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:53.884894 29607 catalog_manager.cc:5719] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 reported cstate change: term changed from 4 to 5, leader changed from 70a127fea97b464a89d03c5848cf6637 (127.28.157.1) to b5d8d89741dd4568a4c7b210a4da72d2 (127.28.157.2). New cstate: current_term: 5 leader_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } health_report { overall_health: UNKNOWN } } }
00:14:54.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:54.382990 30684 consensus_peers.cc:597] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 -> Peer 70a127fea97b464a89d03c5848cf6637 (127.28.157.1:37179): Couldn't send request to peer 70a127fea97b464a89d03c5848cf6637. Status: Network error: Client connection negotiation failed: client connection to 127.28.157.1:37179: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
00:14:54.397 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:14:54.397 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367),70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179),b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)[L]]}]
00:14:54.397 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=12, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=1590), Traces: [0ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [2ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [2ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [10ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [11ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [13ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [13ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [30ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [32ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [35ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [35ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [45ms] refreshing cache from master, [45ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [55ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [56ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [59ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [59ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [186ms] refreshing cache from master, [186ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [191ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [194ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [196ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [196ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [645ms] refreshing cache from master, [646ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [650ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [651ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [653ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [653ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [1585ms] refreshing cache from master, [1585ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [1589ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@1359238655(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@ed4ef8b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@411442383 after retry RPC@653722059) -> (continuation of Deferred@727009250 after retry RPC@823771659) -> (continuation of Deferred@407645932 after retry RPC@1731898996) -> (continuation of Deferred@1937049283 after retry RPC@1091456983) -> (continuation of Deferred@366184379 after retry RPC@646971937), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1105e955 -> wakeup thread Time-limited test -> (continuation of Deferred@411442383 after retry RPC@653722059) -> (continuation of Deferred@727009250 after retry RPC@823771659) -> (continuation of Deferred@407645932 after retry RPC@1731898996) -> (continuation of Deferred@1937049283 after retry RPC@1091456983) -> (continuation of Deferred@366184379 after retry RPC@646971937))) after lookup
00:14:54.398 [DEBUG - kudu-nio-2] (Connection.java:748) [peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)] connecting to peer
00:14:54.399 [DEBUG - kudu-nio-0] (Connection.java:757) [peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)] Successfully connected to peer
00:14:54.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:54.407771 31433 raft_consensus.cc:1275] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 5 FOLLOWER]: Refusing update from remote peer b5d8d89741dd4568a4c7b210a4da72d2: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 5 index: 17. (index mismatch)
00:14:54.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:54.409469 31570 consensus_queue.cc:1048] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 12, Time since last communication: 0.000s
00:14:54.426 [DEBUG - kudu-nio-0] (Negotiator.java:918) Authenticated connection [id: 0xb45cf834, L:/127.0.0.1:33746 - R:/127.28.157.2:45951] using TOKEN/null
00:14:54.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:54.432845 31577 mvcc.cc:204] Tried to move back new op lower bound from 7317478152912240640 to 7317478149466152960. Current Snapshot: MvccSnapshot[applied={T|T < 7317478152912240640}]
00:14:54.501 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.28.157.1:37179
00:14:54.502 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

00:14:54.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:54.503772 29300 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu
00:14:54.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu
00:14:54.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/wal
00:14:54.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/data
00:14:54.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
00:14:54.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
00:14:54.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
00:14:54.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
00:14:54.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
00:14:54.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
00:14:54.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/logs
00:14:54.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/data/info.pb
00:14:54.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
00:14:54.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
00:14:54.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
00:14:54.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
00:14:54.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
00:14:54.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
00:14:54.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
00:14:54.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
00:14:54.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
00:14:54.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
00:14:54.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.28.157.1:37179
00:14:54.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.28.157.1
00:14:54.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.28.157.62:35965,127.28.157.61:44801,127.28.157.60:33393
00:14:54.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=35519
00:14:54.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.28.157.1
00:14:54.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.28.157.20:34671
00:14:54.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
00:14:54.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
00:14:54.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
00:14:54.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:54.872965 31593 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
00:14:54.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:54.873540 31593 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
00:14:54.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:54.874183 31593 flags.cc:432] Enabled unsafe flag: --never_fsync=true
00:14:54.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:54.908515 31593 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
00:14:54.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:54.909497 31593 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.157.1
00:14:54.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:54.948310 31593 tablet_server_runner.cc:78] Tablet server non-default flags:
00:14:54.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/logs
00:14:54.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
00:14:54.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
00:14:54.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
00:14:54.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.28.157.20:34671
00:14:54.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
00:14:54.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
00:14:54.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/data
00:14:54.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/wal
00:14:54.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
00:14:54.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
00:14:54.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.28.157.1:37179
00:14:54.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
00:14:54.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
00:14:54.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
00:14:54.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/data/info.pb
00:14:54.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.28.157.1
00:14:54.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=35519
00:14:54.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.28.157.62:35965,127.28.157.61:44801,127.28.157.60:33393
00:14:54.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
00:14:54.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
00:14:54.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
00:14:54.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
00:14:54.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
00:14:54.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.28.157.1
00:14:54.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
00:14:54.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
00:14:54.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
00:14:54.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision f7ed5b56c096f54e2f926bf94ceb7fc44c0c1329
00:14:54.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
00:14:54.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 11 Aug 2026 23:55:14 UTC on e7f111948823
00:14:54.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 17639
00:14:54.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
00:14:54.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:54.950109 31593 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
00:14:54.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:54.951844 31593 file_cache.cc:504] Constructed file cache file cache with capacity 419430
00:14:54.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:54.965287 31604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
00:14:56.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:54.966879 31605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
00:14:56.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:54.971696 31609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
00:14:56.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:56.129907 31607 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1159 milliseconds
00:14:56.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:56.130044 31593 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
00:14:56.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:56.131309 31593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
00:14:56.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:56.133543 31593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
00:14:56.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:56.135123 31593 hybrid_clock.cc:648] HybridClock initialized: now 1786493696135065 us; error 36 us; skew 500 ppm
00:14:56.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:56.136327 31593 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
00:14:56.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:56.143374 31593 webserver.cc:533] Webserver started at http://127.28.157.1:35519/ using document root <none> and password file <none>
00:14:56.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:56.144541 31593 fs_manager.cc:362] Metadata directory not provided
00:14:56.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:56.144804 31593 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
00:14:56.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:56.153374 31593 fs_manager.cc:714] Time spent opening directory manager: real 0.005s	user 0.006s	sys 0.001s
00:14:56.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:56.159140 31643 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
00:14:56.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:56.160494 31593 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.005s	sys 0.000s
00:14:56.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:56.160977 31593 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/data,/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/wal
00:14:56.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "70a127fea97b464a89d03c5848cf6637"
00:14:56.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 00:14:32 on dist-test-slave-t7g5"
00:14:56.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:56.163257 31593 fs_report.cc:389] FS layout report
00:14:56.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
00:14:56.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/wal
00:14:56.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/wal
00:14:56.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/data/data
00:14:56.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
00:14:56.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
00:14:56.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
00:14:56.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
00:14:56.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
00:14:56.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
00:14:56.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
00:14:56.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
00:14:56.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
00:14:56.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
00:14:56.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
00:14:56.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:56.224388 31593 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
00:14:56.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:56.226146 31593 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
00:14:56.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:56.226676 31593 kserver.cc:163] Server-wide thread pool size limit: 3276
00:14:56.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:56.229777 31593 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
00:14:56.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:56.236033 31656 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
00:14:56.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:56.246083 31593 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
00:14:56.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:56.246387 31593 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.012s	user 0.000s	sys 0.002s
00:14:56.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:56.246697 31593 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
00:14:56.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:56.251852 31593 ts_tablet_manager.cc:616] Registered 1 tablets
00:14:56.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:56.252060 31593 ts_tablet_manager.cc:595] Time spent register tablets: real 0.005s	user 0.000s	sys 0.004s
00:14:56.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:56.252694 31656 tablet_bootstrap.cc:492] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637: Bootstrap starting.
00:14:56.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:56.315886 31656 log.cc:826] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637: Log is configured to *not* fsync() on all Append() calls
00:14:56.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:56.392405 31656 tablet_bootstrap.cc:492] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637: Bootstrap replayed 1/1 log segments. Stats: ops{read=16 overwritten=0 applied=16 ignored=0} inserts{seen=12 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
00:14:56.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:56.393537 31656 tablet_bootstrap.cc:492] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637: Bootstrap complete.
00:14:56.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:56.395582 31656 ts_tablet_manager.cc:1403] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637: Time spent bootstrapping tablet: real 0.143s	user 0.097s	sys 0.043s
00:14:56.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:56.421314 31656 raft_consensus.cc:359] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:14:56.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:56.424818 31656 raft_consensus.cc:740] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70a127fea97b464a89d03c5848cf6637, State: Initialized, Role: FOLLOWER
00:14:56.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:56.426066 31656 consensus_queue.cc:260] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 4.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:14:56.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:56.428901 31593 rpc_server.cc:307] RPC server started. Bound to: 127.28.157.1:37179
00:14:56.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:56.429059 31818 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.157.1:37179 every 8 connection(s)
00:14:56.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:56.430765 31656 ts_tablet_manager.cc:1434] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637: Time spent starting tablet: real 0.035s	user 0.035s	sys 0.001s
00:14:56.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:56.432359 31593 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/data/info.pb
00:14:56.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:56.436141 29300 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu as pid 31593
00:14:56.437 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
00:14:56.471 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 15 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
00:14:56.474 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.28.157.2:45951
00:14:56.474 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

00:14:56.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:56.474869 29300 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu with pid 30644
00:14:56.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:56.500933 31824 heartbeater.cc:344] Connected to a master server at 127.28.157.62:35965
00:14:56.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:56.501498 31824 heartbeater.cc:461] Registering TS with master...
00:14:56.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:56.502977 31824 heartbeater.cc:507] Master 127.28.157.62:35965 requested a full tablet report, sending...
00:14:56.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:56.510520 29360 ts_manager.cc:194] Re-registered known tserver with Master: 70a127fea97b464a89d03c5848cf6637 (127.28.157.1:37179)
00:14:56.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:56.515025 31820 heartbeater.cc:344] Connected to a master server at 127.28.157.60:33393
00:14:56.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:56.515442 31820 heartbeater.cc:461] Registering TS with master...
00:14:56.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:56.516407 31820 heartbeater.cc:507] Master 127.28.157.60:33393 requested a full tablet report, sending...
00:14:56.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:56.518319 31821 heartbeater.cc:344] Connected to a master server at 127.28.157.61:44801
00:14:56.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:56.518635 31821 heartbeater.cc:461] Registering TS with master...
00:14:56.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:56.519409 31821 heartbeater.cc:507] Master 127.28.157.61:44801 requested a full tablet report, sending...
00:14:56.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:56.519472 29607 ts_manager.cc:194] Re-registered known tserver with Master: 70a127fea97b464a89d03c5848cf6637 (127.28.157.1:37179)
00:14:56.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:56.522341 29481 ts_manager.cc:194] Re-registered known tserver with Master: 70a127fea97b464a89d03c5848cf6637 (127.28.157.1:37179)
00:14:56.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:56.522418 29607 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.28.157.1:34599
00:14:56.549 [DEBUG - kudu-nio-0] (Connection.java:239) [peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)] handling channelInactive
00:14:56.549 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
00:14:56.550 [DEBUG - kudu-nio-0] (Connection.java:709) [peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)] cleaning up while in state READY due to: connection closed
00:14:56.550 [DEBUG - Time-limited test] (Connection.java:748) [peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)] connecting to peer
00:14:56.551 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951
00:14:56.551 [DEBUG - kudu-nio-1] (Connection.java:709) [peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)] cleaning up while in state CONNECTING due to: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951
00:14:56.552 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951
00:14:56.577 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:14:56.578 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367),70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179),b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)[L]]}]
00:14:56.578 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=2, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=28), Traces: [0ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [1ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [2ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [22ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [27ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@636680650(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@3d5765f -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@229e2ba0 -> wakeup thread Time-limited test)) after lookup
00:14:56.578 [DEBUG - kudu-nio-2] (Connection.java:748) [peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)] connecting to peer
00:14:56.579 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951
00:14:56.580 [DEBUG - kudu-nio-2] (Connection.java:709) [peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)] cleaning up while in state CONNECTING due to: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951
00:14:56.580 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951
00:14:56.597 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:14:56.597 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367),70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179),b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)[L]]}]
00:14:56.598 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=4, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=47), Traces: [0ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [1ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [2ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [22ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [27ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [28ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [30ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [30ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [42ms] refreshing cache from master, [42ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [47ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@636680650(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@3d5765f -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@426061789 after retry RPC@2135451680), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@229e2ba0 -> wakeup thread Time-limited test -> (continuation of Deferred@426061789 after retry RPC@2135451680))) after lookup
00:14:56.599 [DEBUG - kudu-nio-2] (Connection.java:748) [peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)] connecting to peer
00:14:56.600 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951
00:14:56.600 [DEBUG - kudu-nio-3] (Connection.java:709) [peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)] cleaning up while in state CONNECTING due to: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951
00:14:56.600 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951
00:14:56.617 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:14:56.618 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367),70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179),b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)[L]]}]
00:14:56.618 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=6, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=68), Traces: [0ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [1ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [2ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [22ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [27ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [28ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [30ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [30ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [42ms] refreshing cache from master, [42ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [47ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [48ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [50ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [50ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [62ms] refreshing cache from master, [62ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [66ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@636680650(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@3d5765f -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@426061789 after retry RPC@2135451680) -> (continuation of Deferred@925786507 after retry RPC@194713126), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@229e2ba0 -> wakeup thread Time-limited test -> (continuation of Deferred@426061789 after retry RPC@2135451680) -> (continuation of Deferred@925786507 after retry RPC@194713126))) after lookup
00:14:56.619 [DEBUG - kudu-nio-2] (Connection.java:748) [peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)] connecting to peer
00:14:56.621 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951
00:14:56.621 [DEBUG - kudu-nio-4] (Connection.java:709) [peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)] cleaning up while in state CONNECTING due to: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951
00:14:56.621 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951
00:14:56.717 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:14:56.718 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367),70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179),b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)[L]]}]
00:14:56.718 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=8, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=168), Traces: [0ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [1ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [2ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [22ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [27ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [28ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [30ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [30ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [42ms] refreshing cache from master, [42ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [47ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [48ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [50ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [50ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [62ms] refreshing cache from master, [62ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [66ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [69ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [71ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [71ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [162ms] refreshing cache from master, [162ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [167ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@636680650(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@3d5765f -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@426061789 after retry RPC@2135451680) -> (continuation of Deferred@925786507 after retry RPC@194713126) -> (continuation of Deferred@1363981643 after retry RPC@45462370), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@229e2ba0 -> wakeup thread Time-limited test -> (continuation of Deferred@426061789 after retry RPC@2135451680) -> (continuation of Deferred@925786507 after retry RPC@194713126) -> (continuation of Deferred@1363981643 after retry RPC@45462370))) after lookup
00:14:56.719 [DEBUG - kudu-nio-2] (Connection.java:748) [peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)] connecting to peer
00:14:56.720 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951
00:14:56.721 [DEBUG - kudu-nio-5] (Connection.java:709) [peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)] cleaning up while in state CONNECTING due to: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951
00:14:56.721 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951
00:14:56.897 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:14:56.898 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367),70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179),b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)[L]]}]
00:14:56.898 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=10, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=348), Traces: [0ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [1ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [2ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [22ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [27ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [28ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [30ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [30ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [42ms] refreshing cache from master, [42ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [47ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [48ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [50ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [50ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [62ms] refreshing cache from master, [62ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [66ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [69ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [71ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [71ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [162ms] refreshing cache from master, [162ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [167ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [169ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [171ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [171ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [342ms] refreshing cache from master, [342ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [347ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@636680650(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@3d5765f -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@426061789 after retry RPC@2135451680) -> (continuation of Deferred@925786507 after retry RPC@194713126) -> (continuation of Deferred@1363981643 after retry RPC@45462370) -> (continuation of Deferred@1045337534 after retry RPC@290398134), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@229e2ba0 -> wakeup thread Time-limited test -> (continuation of Deferred@426061789 after retry RPC@2135451680) -> (continuation of Deferred@925786507 after retry RPC@194713126) -> (continuation of Deferred@1363981643 after retry RPC@45462370) -> (continuation of Deferred@1045337534 after retry RPC@290398134))) after lookup
00:14:56.899 [DEBUG - kudu-nio-2] (Connection.java:748) [peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)] connecting to peer
00:14:56.900 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951
00:14:56.900 [DEBUG - kudu-nio-6] (Connection.java:709) [peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)] cleaning up while in state CONNECTING due to: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951
00:14:56.900 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951
00:14:57.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:57.526338 31820 heartbeater.cc:499] Master 127.28.157.60:33393 was elected leader, sending a full tablet report...
00:14:57.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:57.577955 31842 raft_consensus.cc:493] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b5d8d89741dd4568a4c7b210a4da72d2)
00:14:57.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:57.578449 31842 raft_consensus.cc:515] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:14:57.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:57.581007 31842 leader_election.cc:290] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b5d8d89741dd4568a4c7b210a4da72d2 (127.28.157.2:45951), 70a127fea97b464a89d03c5848cf6637 (127.28.157.1:37179)
00:14:57.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:57.584755 31316 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.157.2:45951: connect: Connection refused (error 111)
00:14:57.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:57.585052 31855 raft_consensus.cc:493] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
00:14:57.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:57.585601 31855 raft_consensus.cc:515] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:14:57.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:57.589370 31316 leader_election.cc:336] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer b5d8d89741dd4568a4c7b210a4da72d2 (127.28.157.2:45951): Network error: Client connection negotiation failed: client connection to 127.28.157.2:45951: connect: Connection refused (error 111)
00:14:57.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:57.589843 31855 leader_election.cc:290] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b5d8d89741dd4568a4c7b210a4da72d2 (127.28.157.2:45951), 094dbc44c6eb439083946b5ab0085ad0 (127.28.157.3:42367)
00:14:57.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:57.593118 31652 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.157.2:45951: connect: Connection refused (error 111)
00:14:57.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:57.598155 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f4c1ce5aa4a4c8da912065e2decef6a" candidate_uuid: "094dbc44c6eb439083946b5ab0085ad0" candidate_term: 6 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "70a127fea97b464a89d03c5848cf6637" is_pre_election: true
00:14:57.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:57.599573 31745 raft_consensus.cc:2468] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 094dbc44c6eb439083946b5ab0085ad0 in term 4.
00:14:57.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:57.601297 31652 leader_election.cc:336] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer b5d8d89741dd4568a4c7b210a4da72d2 (127.28.157.2:45951): Network error: Client connection negotiation failed: client connection to 127.28.157.2:45951: connect: Connection refused (error 111)
00:14:57.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:57.602480 31308 leader_election.cc:304] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 094dbc44c6eb439083946b5ab0085ad0, 70a127fea97b464a89d03c5848cf6637; no voters: b5d8d89741dd4568a4c7b210a4da72d2
00:14:57.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:57.603837 31842 raft_consensus.cc:2804] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 5 FOLLOWER]: Leader pre-election won for term 6
00:14:57.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:57.604135 31842 raft_consensus.cc:493] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader b5d8d89741dd4568a4c7b210a4da72d2)
00:14:57.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:57.604517 31842 raft_consensus.cc:3060] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 5 FOLLOWER]: Advancing to term 6
00:14:57.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:57.605981 31433 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f4c1ce5aa4a4c8da912065e2decef6a" candidate_uuid: "70a127fea97b464a89d03c5848cf6637" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "094dbc44c6eb439083946b5ab0085ad0" is_pre_election: true
00:14:57.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:57.610152 31842 raft_consensus.cc:515] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:14:57.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:57.611390 31433 raft_consensus.cc:2368] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 70a127fea97b464a89d03c5848cf6637 for earlier term 5. Current term is 6.
00:14:57.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:57.613483 31842 leader_election.cc:290] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [CANDIDATE]: Term 6 election: Requested vote from peers b5d8d89741dd4568a4c7b210a4da72d2 (127.28.157.2:45951), 70a127fea97b464a89d03c5848cf6637 (127.28.157.1:37179)
00:14:57.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:57.614638 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f4c1ce5aa4a4c8da912065e2decef6a" candidate_uuid: "094dbc44c6eb439083946b5ab0085ad0" candidate_term: 6 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "70a127fea97b464a89d03c5848cf6637"
00:14:57.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:57.615257 31745 raft_consensus.cc:3060] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [term 4 FOLLOWER]: Advancing to term 6
00:14:57.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:57.617163 31649 leader_election.cc:400] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 094dbc44c6eb439083946b5ab0085ad0 (127.28.157.3:42367) with higher term. Message: Invalid argument: T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 70a127fea97b464a89d03c5848cf6637 for earlier term 5. Current term is 6.
00:14:57.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:57.617466 31649 leader_election.cc:403] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
00:14:57.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:57.618330 31316 leader_election.cc:336] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer b5d8d89741dd4568a4c7b210a4da72d2 (127.28.157.2:45951): Network error: Client connection negotiation failed: client connection to 127.28.157.2:45951: connect: Connection refused (error 111)
00:14:57.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:57.623195 31745 raft_consensus.cc:2468] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 094dbc44c6eb439083946b5ab0085ad0 in term 6.
00:14:57.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:57.623823 31855 raft_consensus.cc:2749] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 094dbc44c6eb439083946b5ab0085ad0 (127.28.157.3:42367) with higher term. Message: Invalid argument: T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 70a127fea97b464a89d03c5848cf6637 for earlier term 5. Current term is 6.
00:14:57.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:57.624569 31308 leader_election.cc:304] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 094dbc44c6eb439083946b5ab0085ad0, 70a127fea97b464a89d03c5848cf6637; no voters: b5d8d89741dd4568a4c7b210a4da72d2
00:14:57.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:57.625329 31842 raft_consensus.cc:2804] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 6 FOLLOWER]: Leader election won for term 6
00:14:57.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:57.627450 31842 raft_consensus.cc:697] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 6 LEADER]: Becoming Leader. State: Replica: 094dbc44c6eb439083946b5ab0085ad0, State: Running, Role: LEADER
00:14:57.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:57.628600 31842 consensus_queue.cc:237] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 5.20, Last appended by leader: 20, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:14:57.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:57.639724 29607 catalog_manager.cc:5719] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 reported cstate change: term changed from 5 to 6, leader changed from b5d8d89741dd4568a4c7b210a4da72d2 (127.28.157.2) to 094dbc44c6eb439083946b5ab0085ad0 (127.28.157.3). New cstate: current_term: 6 leader_uuid: "094dbc44c6eb439083946b5ab0085ad0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } health_report { overall_health: UNKNOWN } } }
00:14:58.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:58.133109 31745 raft_consensus.cc:1275] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [term 6 FOLLOWER]: Refusing update from remote peer 094dbc44c6eb439083946b5ab0085ad0: Log matching property violated. Preceding OpId in replica: term: 4 index: 16. Preceding OpId from leader: term: 6 index: 21. (index mismatch)
00:14:58.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:58.134608 31842 consensus_queue.cc:1048] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 16, Time since last communication: 0.000s
00:14:58.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:58.174813 31316 consensus_peers.cc:597] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 -> Peer b5d8d89741dd4568a4c7b210a4da72d2 (127.28.157.2:45951): Couldn't send request to peer b5d8d89741dd4568a4c7b210a4da72d2. Status: Network error: Client connection negotiation failed: client connection to 127.28.157.2:45951: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
00:14:58.577 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:14:58.578 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)[L],70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179),b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)]}]
00:14:58.578 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=12, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=2028), Traces: [0ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [1ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [2ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [22ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [27ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [28ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [30ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [30ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [42ms] refreshing cache from master, [42ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [47ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [48ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [50ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [50ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [62ms] refreshing cache from master, [62ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [66ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [69ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [71ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [71ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [162ms] refreshing cache from master, [162ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [167ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [169ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [171ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [171ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [342ms] refreshing cache from master, [342ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [347ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [349ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [350ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [350ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [2022ms] refreshing cache from master, [2022ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [2027ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@636680650(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@3d5765f -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@426061789 after retry RPC@2135451680) -> (continuation of Deferred@925786507 after retry RPC@194713126) -> (continuation of Deferred@1363981643 after retry RPC@45462370) -> (continuation of Deferred@1045337534 after retry RPC@290398134) -> (continuation of Deferred@1602518362 after retry RPC@912227251), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@229e2ba0 -> wakeup thread Time-limited test -> (continuation of Deferred@426061789 after retry RPC@2135451680) -> (continuation of Deferred@925786507 after retry RPC@194713126) -> (continuation of Deferred@1363981643 after retry RPC@45462370) -> (continuation of Deferred@1045337534 after retry RPC@290398134) -> (continuation of Deferred@1602518362 after retry RPC@912227251))) after lookup
00:14:58.579 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)] connecting to peer
00:14:58.581 [DEBUG - kudu-nio-7] (Connection.java:757) [peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)] Successfully connected to peer
00:14:58.615 [DEBUG - kudu-nio-7] (Negotiator.java:918) Authenticated connection [id: 0x7a3070ce, L:/127.0.0.1:39106 - R:/127.28.157.3:42367] using TOKEN/null
00:14:58.704 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.28.157.2:45951
00:14:58.706 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

00:14:58.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:58.707738 29300 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu
00:14:58.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu
00:14:58.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-1/wal
00:14:58.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-1/data
00:14:58.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
00:14:58.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
00:14:58.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
00:14:58.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
00:14:58.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
00:14:58.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
00:14:58.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-1/logs
00:14:58.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-1/data/info.pb
00:14:58.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
00:14:58.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
00:14:58.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
00:14:58.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
00:14:58.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
00:14:58.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
00:14:58.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
00:14:58.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
00:14:58.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
00:14:58.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
00:14:58.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.28.157.2:45951
00:14:58.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.28.157.2
00:14:58.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.28.157.62:35965,127.28.157.61:44801,127.28.157.60:33393
00:14:58.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40801
00:14:58.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.28.157.2
00:14:58.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.28.157.20:34671
00:14:58.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
00:14:58.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
00:14:58.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
00:14:59.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:59.063915 31907 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
00:14:59.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:59.064430 31907 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
00:14:59.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:59.064906 31907 flags.cc:432] Enabled unsafe flag: --never_fsync=true
00:14:59.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:59.099673 31907 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
00:14:59.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:59.100598 31907 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.157.2
00:14:59.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:59.140494 31907 tablet_server_runner.cc:78] Tablet server non-default flags:
00:14:59.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-1/logs
00:14:59.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
00:14:59.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
00:14:59.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
00:14:59.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.28.157.20:34671
00:14:59.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
00:14:59.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
00:14:59.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-1/data
00:14:59.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-1/wal
00:14:59.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
00:14:59.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
00:14:59.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.28.157.2:45951
00:14:59.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
00:14:59.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
00:14:59.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
00:14:59.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-1/data/info.pb
00:14:59.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.28.157.2
00:14:59.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40801
00:14:59.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.28.157.62:35965,127.28.157.61:44801,127.28.157.60:33393
00:14:59.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
00:14:59.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
00:14:59.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
00:14:59.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
00:14:59.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
00:14:59.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.28.157.2
00:14:59.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
00:14:59.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
00:14:59.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
00:14:59.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision f7ed5b56c096f54e2f926bf94ceb7fc44c0c1329
00:14:59.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
00:14:59.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 11 Aug 2026 23:55:14 UTC on e7f111948823
00:14:59.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 17639
00:14:59.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
00:14:59.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:59.141906 31907 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
00:14:59.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:14:59.143736 31907 file_cache.cc:504] Constructed file cache file cache with capacity 419430
00:14:59.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:59.158044 31917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
00:15:00.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:00.530112 31316 consensus_peers.cc:597] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 -> Peer b5d8d89741dd4568a4c7b210a4da72d2 (127.28.157.2:45951): Couldn't send request to peer b5d8d89741dd4568a4c7b210a4da72d2. Status: Network error: Client connection negotiation failed: client connection to 127.28.157.2:45951: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
00:15:00.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:14:59.159489 31918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
00:15:00.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:00.557385 31921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
00:15:00.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:00.563443 31920 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1395 milliseconds
00:15:00.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:00.563788 31907 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
00:15:00.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:00.566581 31907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
00:15:00.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:00.572805 31907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
00:15:00.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:00.574815 31907 hybrid_clock.cc:648] HybridClock initialized: now 1786493700574669 us; error 100 us; skew 500 ppm
00:15:00.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:00.577126 31907 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
00:15:00.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:00.591311 31907 webserver.cc:533] Webserver started at http://127.28.157.2:40801/ using document root <none> and password file <none>
00:15:00.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:00.593464 31907 fs_manager.cc:362] Metadata directory not provided
00:15:00.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:00.594199 31907 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
00:15:00.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:00.614151 31907 fs_manager.cc:714] Time spent opening directory manager: real 0.013s	user 0.006s	sys 0.008s
00:15:00.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:00.627208 31944 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
00:15:00.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:00.631035 31907 fs_manager.cc:730] Time spent opening block manager: real 0.010s	user 0.006s	sys 0.003s
00:15:00.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:00.632579 31907 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-1/data,/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-1/wal
00:15:00.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "b5d8d89741dd4568a4c7b210a4da72d2"
00:15:00.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 00:14:34 on dist-test-slave-t7g5"
00:15:00.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:00.639988 31907 fs_report.cc:389] FS layout report
00:15:00.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
00:15:00.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-1/wal
00:15:00.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-1/wal
00:15:00.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-1/data/data
00:15:00.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
00:15:00.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
00:15:00.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
00:15:00.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
00:15:00.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
00:15:00.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
00:15:00.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
00:15:00.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
00:15:00.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
00:15:00.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
00:15:00.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
00:15:00.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:00.783746 31907 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
00:15:00.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:00.787284 31907 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
00:15:00.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:00.788285 31907 kserver.cc:163] Server-wide thread pool size limit: 3276
00:15:00.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:00.796969 31907 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
00:15:00.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:00.811622 31958 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
00:15:00.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:00.832057 31907 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
00:15:00.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:00.832639 31907 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.024s	user 0.000s	sys 0.004s
00:15:00.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:00.833146 31907 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
00:15:00.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:00.843474 31907 ts_tablet_manager.cc:616] Registered 1 tablets
00:15:00.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:00.843852 31907 ts_tablet_manager.cc:595] Time spent register tablets: real 0.011s	user 0.010s	sys 0.000s
00:15:00.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:00.845078 31958 tablet_bootstrap.cc:492] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2: Bootstrap starting.
00:15:00.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:00.982820 31958 log.cc:826] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2: Log is configured to *not* fsync() on all Append() calls
00:15:01.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:01.074322 31958 tablet_bootstrap.cc:492] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2: Bootstrap replayed 1/1 log segments. Stats: ops{read=20 overwritten=0 applied=20 ignored=0} inserts{seen=15 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
00:15:01.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:01.075572 31958 tablet_bootstrap.cc:492] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2: Bootstrap complete.
00:15:01.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:01.077584 31958 ts_tablet_manager.cc:1403] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2: Time spent bootstrapping tablet: real 0.234s	user 0.166s	sys 0.050s
00:15:01.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:01.099357 31907 rpc_server.cc:307] RPC server started. Bound to: 127.28.157.2:45951
00:15:01.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:01.099584 32141 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.157.2:45951 every 8 connection(s)
00:15:01.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:01.103196 31907 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-1/data/info.pb
00:15:01.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:01.103075 31958 raft_consensus.cc:359] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:15:01.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:01.106398 31958 raft_consensus.cc:740] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5d8d89741dd4568a4c7b210a4da72d2, State: Initialized, Role: FOLLOWER
00:15:01.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:01.107937 31958 consensus_queue.cc:260] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 20, Last appended: 5.20, Last appended by leader: 20, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:15:01.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:01.110237 29300 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu as pid 31907
00:15:01.111 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
00:15:01.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:01.136339 31958 ts_tablet_manager.cc:1434] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2: Time spent starting tablet: real 0.058s	user 0.037s	sys 0.005s
00:15:01.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:01.163203 32143 heartbeater.cc:344] Connected to a master server at 127.28.157.60:33393
00:15:01.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:01.163765 32143 heartbeater.cc:461] Registering TS with master...
00:15:01.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:01.164167 32144 heartbeater.cc:344] Connected to a master server at 127.28.157.61:44801
00:15:01.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:01.164547 32144 heartbeater.cc:461] Registering TS with master...
00:15:01.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:01.165215 32143 heartbeater.cc:507] Master 127.28.157.60:33393 requested a full tablet report, sending...
00:15:01.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:01.165905 32144 heartbeater.cc:507] Master 127.28.157.61:44801 requested a full tablet report, sending...
00:15:01.169 [DEBUG - kudu-nio-7] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 18 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
00:15:01.170 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.28.157.3:42367
00:15:01.171 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

00:15:01.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:01.170236 29607 ts_manager.cc:194] Re-registered known tserver with Master: b5d8d89741dd4568a4c7b210a4da72d2 (127.28.157.2:45951)
00:15:01.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:01.170249 29481 ts_manager.cc:194] Re-registered known tserver with Master: b5d8d89741dd4568a4c7b210a4da72d2 (127.28.157.2:45951)
00:15:01.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:01.171460 29300 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu with pid 31266
00:15:01.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:01.176156 32145 heartbeater.cc:344] Connected to a master server at 127.28.157.62:35965
00:15:01.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:01.176551 32145 heartbeater.cc:461] Registering TS with master...
00:15:01.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:01.177469 32145 heartbeater.cc:507] Master 127.28.157.62:35965 requested a full tablet report, sending...
00:15:01.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:01.177692 29607 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.28.157.2:48871
00:15:01.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:01.183988 32143 heartbeater.cc:499] Master 127.28.157.60:33393 was elected leader, sending a full tablet report...
00:15:01.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:01.187060 29360 ts_manager.cc:194] Re-registered known tserver with Master: b5d8d89741dd4568a4c7b210a4da72d2 (127.28.157.2:45951)
00:15:01.213 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
00:15:01.214 [DEBUG - kudu-nio-7] (Connection.java:239) [peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)] handling channelInactive
00:15:01.214 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)] cleaning up while in state READY due to: connection closed
00:15:01.233 [DEBUG - pool-2-thread-1] (Connection.java:748) [peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)] connecting to peer
00:15:01.234 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367
00:15:01.234 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)] cleaning up while in state CONNECTING due to: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367
00:15:01.234 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367
00:15:01.256 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:15:01.257 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)[L],70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179),b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)]}]
00:15:01.257 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=3, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=43), Traces: [0ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [0ms] delaying RPC due to: Illegal state: connection is terminated, [18ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [20ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [20ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [42ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@1637425226(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@2c565d29 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@61061ce9 -> wakeup thread Time-limited test)) after lookup
00:15:01.258 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)] connecting to peer
00:15:01.259 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367
00:15:01.259 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)] cleaning up while in state CONNECTING due to: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367
00:15:01.259 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367
00:15:01.276 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:15:01.277 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)[L],70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179),b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)]}]
00:15:01.277 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=5, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=63), Traces: [0ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [0ms] delaying RPC due to: Illegal state: connection is terminated, [18ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [20ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [20ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [42ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [43ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [45ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [45ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [58ms] refreshing cache from master, [58ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [62ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@1637425226(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@2c565d29 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@358436805 after retry RPC@1148053715), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@61061ce9 -> wakeup thread Time-limited test -> (continuation of Deferred@358436805 after retry RPC@1148053715))) after lookup
00:15:01.278 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)] connecting to peer
00:15:01.279 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367
00:15:01.279 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)] cleaning up while in state CONNECTING due to: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367
00:15:01.279 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367
00:15:01.297 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:15:01.297 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)[L],70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179),b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)]}]
00:15:01.298 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=7, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=84), Traces: [0ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [0ms] delaying RPC due to: Illegal state: connection is terminated, [18ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [20ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [20ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [42ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [43ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [45ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [45ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [58ms] refreshing cache from master, [58ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [62ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [64ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [65ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [65ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [78ms] refreshing cache from master, [79ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [82ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@1637425226(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@2c565d29 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@358436805 after retry RPC@1148053715) -> (continuation of Deferred@39648695 after retry RPC@1867427663), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@61061ce9 -> wakeup thread Time-limited test -> (continuation of Deferred@358436805 after retry RPC@1148053715) -> (continuation of Deferred@39648695 after retry RPC@1867427663))) after lookup
00:15:01.298 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)] connecting to peer
00:15:01.299 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367
00:15:01.300 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)] cleaning up while in state CONNECTING due to: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367
00:15:01.300 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367
00:15:01.438 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:15:01.438 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)[L],70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179),b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)]}]
00:15:01.439 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=9, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=225), Traces: [0ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [0ms] delaying RPC due to: Illegal state: connection is terminated, [18ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [20ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [20ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [42ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [43ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [45ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [45ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [58ms] refreshing cache from master, [58ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [62ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [64ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [65ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [65ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [78ms] refreshing cache from master, [79ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [82ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [84ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [86ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [86ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [218ms] refreshing cache from master, [218ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [224ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@1637425226(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@2c565d29 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@358436805 after retry RPC@1148053715) -> (continuation of Deferred@39648695 after retry RPC@1867427663) -> (continuation of Deferred@593659611 after retry RPC@961311023), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@61061ce9 -> wakeup thread Time-limited test -> (continuation of Deferred@358436805 after retry RPC@1148053715) -> (continuation of Deferred@39648695 after retry RPC@1867427663) -> (continuation of Deferred@593659611 after retry RPC@961311023))) after lookup
00:15:01.440 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)] connecting to peer
00:15:01.441 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367
00:15:01.441 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)] cleaning up while in state CONNECTING due to: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367
00:15:01.441 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367
00:15:02.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:02.301285 32164 raft_consensus.cc:493] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 094dbc44c6eb439083946b5ab0085ad0)
00:15:02.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:02.301798 32164 raft_consensus.cc:515] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:15:02.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:02.303400 32164 leader_election.cc:290] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b5d8d89741dd4568a4c7b210a4da72d2 (127.28.157.2:45951), 094dbc44c6eb439083946b5ab0085ad0 (127.28.157.3:42367)
00:15:02.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:02.311688 31649 leader_election.cc:336] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 094dbc44c6eb439083946b5ab0085ad0 (127.28.157.3:42367): Network error: Client connection negotiation failed: client connection to 127.28.157.3:42367: connect: Connection refused (error 111)
00:15:02.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:02.315995 32068 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f4c1ce5aa4a4c8da912065e2decef6a" candidate_uuid: "70a127fea97b464a89d03c5848cf6637" candidate_term: 7 candidate_status { last_received { term: 6 index: 24 } } ignore_live_leader: false dest_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" is_pre_election: true
00:15:02.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:02.316790 32068 raft_consensus.cc:2468] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70a127fea97b464a89d03c5848cf6637 in term 5.
00:15:02.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:02.318145 31652 leader_election.cc:304] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 70a127fea97b464a89d03c5848cf6637, b5d8d89741dd4568a4c7b210a4da72d2; no voters: 094dbc44c6eb439083946b5ab0085ad0
00:15:02.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:02.318923 32164 raft_consensus.cc:2804] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [term 6 FOLLOWER]: Leader pre-election won for term 7
00:15:02.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:02.319191 32164 raft_consensus.cc:493] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 094dbc44c6eb439083946b5ab0085ad0)
00:15:02.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:02.319404 32164 raft_consensus.cc:3060] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [term 6 FOLLOWER]: Advancing to term 7
00:15:02.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:02.324956 32164 raft_consensus.cc:515] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:15:02.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:02.326769 32164 leader_election.cc:290] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [CANDIDATE]: Term 7 election: Requested vote from peers b5d8d89741dd4568a4c7b210a4da72d2 (127.28.157.2:45951), 094dbc44c6eb439083946b5ab0085ad0 (127.28.157.3:42367)
00:15:02.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:02.328400 32068 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f4c1ce5aa4a4c8da912065e2decef6a" candidate_uuid: "70a127fea97b464a89d03c5848cf6637" candidate_term: 7 candidate_status { last_received { term: 6 index: 24 } } ignore_live_leader: false dest_uuid: "b5d8d89741dd4568a4c7b210a4da72d2"
00:15:02.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:02.329034 32068 raft_consensus.cc:3060] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [term 5 FOLLOWER]: Advancing to term 7
00:15:02.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:02.331270 31649 leader_election.cc:336] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 094dbc44c6eb439083946b5ab0085ad0 (127.28.157.3:42367): Network error: Client connection negotiation failed: client connection to 127.28.157.3:42367: connect: Connection refused (error 111)
00:15:02.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:02.336035 32068 raft_consensus.cc:2468] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70a127fea97b464a89d03c5848cf6637 in term 7.
00:15:02.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:02.337205 31652 leader_election.cc:304] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 70a127fea97b464a89d03c5848cf6637, b5d8d89741dd4568a4c7b210a4da72d2; no voters: 094dbc44c6eb439083946b5ab0085ad0
00:15:02.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:02.338052 32164 raft_consensus.cc:2804] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [term 7 FOLLOWER]: Leader election won for term 7
00:15:02.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:02.339238 32164 raft_consensus.cc:697] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [term 7 LEADER]: Becoming Leader. State: Replica: 70a127fea97b464a89d03c5848cf6637, State: Running, Role: LEADER
00:15:02.341 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:02.341032 32164 consensus_queue.cc:237] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 6.24, Last appended by leader: 24, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:15:02.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:02.351293 29607 catalog_manager.cc:5719] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 reported cstate change: term changed from 6 to 7, leader changed from 094dbc44c6eb439083946b5ab0085ad0 (127.28.157.3) to 70a127fea97b464a89d03c5848cf6637 (127.28.157.1). New cstate: current_term: 7 leader_uuid: "70a127fea97b464a89d03c5848cf6637" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } health_report { overall_health: HEALTHY } } }
00:15:02.356 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:15:02.357 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)[L],70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179),b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)]}]
00:15:02.357 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=11, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=1143), Traces: [0ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [0ms] delaying RPC due to: Illegal state: connection is terminated, [18ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [20ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [20ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [42ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [43ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [45ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [45ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [58ms] refreshing cache from master, [58ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [62ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [64ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [65ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [65ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [78ms] refreshing cache from master, [79ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [82ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [84ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [86ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [86ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [218ms] refreshing cache from master, [218ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [224ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [225ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [227ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [227ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [1137ms] refreshing cache from master, [1138ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [1142ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@1637425226(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@2c565d29 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@358436805 after retry RPC@1148053715) -> (continuation of Deferred@39648695 after retry RPC@1867427663) -> (continuation of Deferred@593659611 after retry RPC@961311023) -> (continuation of Deferred@959611531 after retry RPC@1663912331), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@61061ce9 -> wakeup thread Time-limited test -> (continuation of Deferred@358436805 after retry RPC@1148053715) -> (continuation of Deferred@39648695 after retry RPC@1867427663) -> (continuation of Deferred@593659611 after retry RPC@961311023) -> (continuation of Deferred@959611531 after retry RPC@1663912331))) after lookup
00:15:02.358 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)] connecting to peer
00:15:02.362 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367
00:15:02.362 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)] cleaning up while in state CONNECTING due to: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367
00:15:02.362 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367
00:15:02.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:02.828728 31649 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.157.3:42367: connect: Connection refused (error 111) [suppressed 2 similar messages]
00:15:02.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:02.831274 31649 consensus_peers.cc:597] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 -> Peer 094dbc44c6eb439083946b5ab0085ad0 (127.28.157.3:42367): Couldn't send request to peer 094dbc44c6eb439083946b5ab0085ad0. Status: Network error: Client connection negotiation failed: client connection to 127.28.157.3:42367: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
00:15:02.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:02.864789 32068 raft_consensus.cc:1275] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [term 7 FOLLOWER]: Refusing update from remote peer 70a127fea97b464a89d03c5848cf6637: Log matching property violated. Preceding OpId in replica: term: 5 index: 20. Preceding OpId from leader: term: 7 index: 25. (index mismatch)
00:15:02.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:02.867017 32164 consensus_queue.cc:1048] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 20, Time since last communication: 0.000s
00:15:03.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:03.438472 32136 debug-util.cc:398] Leaking SignalData structure 0x7b08000ba0e0 after lost signal to thread 31908
00:15:03.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:03.439702 32136 debug-util.cc:398] Leaking SignalData structure 0x7b08000bad20 after lost signal to thread 32141
00:15:05.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:05.194821 31649 consensus_peers.cc:597] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 -> Peer 094dbc44c6eb439083946b5ab0085ad0 (127.28.157.3:42367): Couldn't send request to peer 094dbc44c6eb439083946b5ab0085ad0. Status: Network error: Client connection negotiation failed: client connection to 127.28.157.3:42367: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
00:15:05.797 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:15:05.797 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367),70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)[L],b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)]}]
00:15:05.798 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=13, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=4583), Traces: [0ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [0ms] delaying RPC due to: Illegal state: connection is terminated, [18ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [20ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [20ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [42ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [43ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [45ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [45ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [58ms] refreshing cache from master, [58ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [62ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [64ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [65ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [65ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [78ms] refreshing cache from master, [79ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [82ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [84ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [86ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [86ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [218ms] refreshing cache from master, [218ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [224ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [225ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [227ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [227ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [1137ms] refreshing cache from master, [1138ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [1142ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [1144ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [1148ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [1148ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [4578ms] refreshing cache from master, [4578ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [4582ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@1637425226(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@2c565d29 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@358436805 after retry RPC@1148053715) -> (continuation of Deferred@39648695 after retry RPC@1867427663) -> (continuation of Deferred@593659611 after retry RPC@961311023) -> (continuation of Deferred@959611531 after retry RPC@1663912331) -> (continuation of Deferred@478453037 after retry RPC@359460391), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@61061ce9 -> wakeup thread Time-limited test -> (continuation of Deferred@358436805 after retry RPC@1148053715) -> (continuation of Deferred@39648695 after retry RPC@1867427663) -> (continuation of Deferred@593659611 after retry RPC@961311023) -> (continuation of Deferred@959611531 after retry RPC@1663912331) -> (continuation of Deferred@478453037 after retry RPC@359460391))) after lookup
00:15:05.798 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] connecting to peer
00:15:05.800 [DEBUG - kudu-nio-6] (Connection.java:757) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] Successfully connected to peer
00:15:05.828 [DEBUG - kudu-nio-6] (Negotiator.java:918) Authenticated connection [id: 0xbd003397, L:/127.0.0.1:39456 - R:/127.28.157.1:37179] using TOKEN/null
00:15:05.883 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.28.157.3:42367
00:15:05.884 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

00:15:05.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:05.886132 29300 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu
00:15:05.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu
00:15:05.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-2/wal
00:15:05.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-2/data
00:15:05.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
00:15:05.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
00:15:05.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
00:15:05.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
00:15:05.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
00:15:05.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
00:15:05.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-2/logs
00:15:05.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-2/data/info.pb
00:15:05.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
00:15:05.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
00:15:05.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
00:15:05.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
00:15:05.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
00:15:05.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
00:15:05.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
00:15:05.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
00:15:05.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
00:15:05.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
00:15:05.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.28.157.3:42367
00:15:05.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.28.157.3
00:15:05.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.28.157.62:35965,127.28.157.61:44801,127.28.157.60:33393
00:15:05.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=36411
00:15:05.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.28.157.3
00:15:05.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.28.157.20:34671
00:15:05.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
00:15:05.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
00:15:05.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
00:15:06.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:06.234018 32236 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
00:15:06.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:06.235044 32236 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
00:15:06.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:06.235898 32236 flags.cc:432] Enabled unsafe flag: --never_fsync=true
00:15:06.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:06.283522 32236 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
00:15:06.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:06.284417 32236 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.157.3
00:15:06.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:06.326061 32236 tablet_server_runner.cc:78] Tablet server non-default flags:
00:15:06.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-2/logs
00:15:06.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
00:15:06.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
00:15:06.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
00:15:06.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.28.157.20:34671
00:15:06.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
00:15:06.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
00:15:06.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-2/data
00:15:06.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-2/wal
00:15:06.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
00:15:06.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
00:15:06.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.28.157.3:42367
00:15:06.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
00:15:06.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
00:15:06.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
00:15:06.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-2/data/info.pb
00:15:06.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.28.157.3
00:15:06.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=36411
00:15:06.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.28.157.62:35965,127.28.157.61:44801,127.28.157.60:33393
00:15:06.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
00:15:06.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
00:15:06.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
00:15:06.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
00:15:06.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
00:15:06.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.28.157.3
00:15:06.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
00:15:06.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
00:15:06.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
00:15:06.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision f7ed5b56c096f54e2f926bf94ceb7fc44c0c1329
00:15:06.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
00:15:06.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 11 Aug 2026 23:55:14 UTC on e7f111948823
00:15:06.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 17639
00:15:06.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
00:15:06.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:06.327386 32236 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
00:15:06.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:06.329190 32236 file_cache.cc:504] Constructed file cache file cache with capacity 419430
00:15:06.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:06.343788 32243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
00:15:07.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:07.837005 31649 consensus_peers.cc:597] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 -> Peer 094dbc44c6eb439083946b5ab0085ad0 (127.28.157.3:42367): Couldn't send request to peer 094dbc44c6eb439083946b5ab0085ad0. Status: Network error: Client connection negotiation failed: client connection to 127.28.157.3:42367: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
00:15:07.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:07.747479 32241 debug-util.cc:398] Leaking SignalData structure 0x7b080002ede0 after lost signal to thread 32236
00:15:07.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:07.844197 32241 kernel_stack_watchdog.cc:198] Thread 32236 stuck at /home/jenkins-slave/workspace/build_and_test@4/src/kudu/util/thread.cc:642 for 401ms:
00:15:07.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Kernel stack:
00:15:07.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) (could not read kernel stack)
00:15:07.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) User stack:
00:15:07.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) <Timed out: thread did not respond: maybe it is blocking signals>
00:15:07.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:06.344156 32244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
00:15:07.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:07.844887 32236 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.501s	user 0.619s	sys 0.829s
00:15:07.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:07.845257 32236 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.502s	user 0.619s	sys 0.829s
00:15:07.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:07.846802 32249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
00:15:07.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:07.848697 32245 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1500 milliseconds
00:15:07.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:07.848716 32236 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
00:15:07.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:07.850055 32236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
00:15:07.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:07.852319 32236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
00:15:07.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:07.853793 32236 hybrid_clock.cc:648] HybridClock initialized: now 1786493707853760 us; error 43 us; skew 500 ppm
00:15:07.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:07.854683 32236 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
00:15:07.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:07.861886 32236 webserver.cc:533] Webserver started at http://127.28.157.3:36411/ using document root <none> and password file <none>
00:15:07.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:07.862895 32236 fs_manager.cc:362] Metadata directory not provided
00:15:07.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:07.863125 32236 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
00:15:07.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:07.871768 32236 fs_manager.cc:714] Time spent opening directory manager: real 0.006s	user 0.005s	sys 0.000s
00:15:07.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:07.877401 32271 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
00:15:07.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:07.878919 32236 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.003s	sys 0.000s
00:15:07.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:07.879323 32236 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-2/data,/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-2/wal
00:15:07.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "094dbc44c6eb439083946b5ab0085ad0"
00:15:07.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 00:14:36 on dist-test-slave-t7g5"
00:15:07.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:07.881482 32236 fs_report.cc:389] FS layout report
00:15:07.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
00:15:07.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-2/wal
00:15:07.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-2/wal
00:15:07.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-2/data/data
00:15:07.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
00:15:07.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
00:15:07.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
00:15:07.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
00:15:07.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
00:15:07.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
00:15:07.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
00:15:07.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
00:15:07.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
00:15:07.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
00:15:07.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
00:15:07.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:07.934243 32236 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
00:15:07.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:07.936260 32236 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
00:15:07.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:07.936810 32236 kserver.cc:163] Server-wide thread pool size limit: 3276
00:15:07.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:07.940801 32236 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
00:15:07.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:07.947343 32278 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
00:15:07.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:07.960678 32236 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
00:15:07.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:07.961047 32236 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.015s	user 0.000s	sys 0.002s
00:15:07.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:07.961386 32236 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
00:15:07.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:07.967368 32236 ts_tablet_manager.cc:616] Registered 1 tablets
00:15:07.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:07.967577 32236 ts_tablet_manager.cc:595] Time spent register tablets: real 0.006s	user 0.005s	sys 0.000s
00:15:07.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:07.968389 32278 tablet_bootstrap.cc:492] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0: Bootstrap starting.
00:15:08.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:08.036234 32278 log.cc:826] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0: Log is configured to *not* fsync() on all Append() calls
00:15:08.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:08.128685 32278 tablet_bootstrap.cc:492] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0: Bootstrap replayed 1/1 log segments. Stats: ops{read=24 overwritten=0 applied=24 ignored=0} inserts{seen=18 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
00:15:08.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:08.129781 32278 tablet_bootstrap.cc:492] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0: Bootstrap complete.
00:15:08.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:08.131634 32278 ts_tablet_manager.cc:1403] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0: Time spent bootstrapping tablet: real 0.164s	user 0.128s	sys 0.030s
00:15:08.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:08.154052 32236 rpc_server.cc:307] RPC server started. Bound to: 127.28.157.3:42367
00:15:08.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:08.154181 32471 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.157.3:42367 every 8 connection(s)
00:15:08.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:08.153015 32278 raft_consensus.cc:359] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 6 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:15:08.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:08.155900 32278 raft_consensus.cc:740] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 094dbc44c6eb439083946b5ab0085ad0, State: Initialized, Role: FOLLOWER
00:15:08.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:08.157028 32236 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-2/data/info.pb
00:15:08.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:08.156771 32278 consensus_queue.cc:260] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 24, Last appended: 6.24, Last appended by leader: 24, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:15:08.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:08.159776 29300 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu as pid 32236
00:15:08.161 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
00:15:08.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:08.170958 32278 ts_tablet_manager.cc:1434] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0: Time spent starting tablet: real 0.039s	user 0.030s	sys 0.009s
00:15:08.184 [DEBUG - kudu-nio-6] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 21 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
00:15:08.185 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.28.157.1:37179
00:15:08.186 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

00:15:08.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:08.186617 29300 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu with pid 31593
00:15:08.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:08.235320 32476 heartbeater.cc:344] Connected to a master server at 127.28.157.60:33393
00:15:08.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:08.235989 32476 heartbeater.cc:461] Registering TS with master...
00:15:08.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:08.237591 32476 heartbeater.cc:507] Master 127.28.157.60:33393 requested a full tablet report, sending...
00:15:08.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:08.244284 29607 ts_manager.cc:194] Re-registered known tserver with Master: 094dbc44c6eb439083946b5ab0085ad0 (127.28.157.3:42367)
00:15:08.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:08.248891 29607 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.28.157.3:38853
00:15:08.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:08.256374 32476 heartbeater.cc:499] Master 127.28.157.60:33393 was elected leader, sending a full tablet report...
00:15:08.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:08.257442 32483 heartbeater.cc:344] Connected to a master server at 127.28.157.62:35965
00:15:08.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:08.257856 32483 heartbeater.cc:461] Registering TS with master...
00:15:08.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:08.258496 32483 heartbeater.cc:507] Master 127.28.157.62:35965 requested a full tablet report, sending...
00:15:08.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:08.262529 29360 ts_manager.cc:194] Re-registered known tserver with Master: 094dbc44c6eb439083946b5ab0085ad0 (127.28.157.3:42367)
00:15:08.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:08.273801 32477 heartbeater.cc:344] Connected to a master server at 127.28.157.61:44801
00:15:08.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:08.274168 32477 heartbeater.cc:461] Registering TS with master...
00:15:08.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:08.274794 32477 heartbeater.cc:507] Master 127.28.157.61:44801 requested a full tablet report, sending...
00:15:08.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:08.278225 29481 ts_manager.cc:194] Re-registered known tserver with Master: 094dbc44c6eb439083946b5ab0085ad0 (127.28.157.3:42367)
00:15:08.283 [DEBUG - kudu-nio-6] (Connection.java:239) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] handling channelInactive
00:15:08.283 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] cleaning up while in state READY due to: connection closed
00:15:08.284 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
00:15:08.286 [DEBUG - Time-limited test] (Connection.java:748) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] connecting to peer
00:15:08.287 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:15:08.287 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] cleaning up while in state CONNECTING due to: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:15:08.288 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:15:08.296 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:15:08.296 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367),70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)[L],b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)]}]
00:15:08.296 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=2, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=11), Traces: [0ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [3ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [3ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [11ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@1179364570(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@34a46fe2 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@8455b86 -> wakeup thread Time-limited test)) after lookup
00:15:08.297 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] connecting to peer
00:15:08.298 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:15:08.298 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] cleaning up while in state CONNECTING due to: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:15:08.298 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:15:08.317 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:15:08.317 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367),70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)[L],b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)]}]
00:15:08.317 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=4, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=32), Traces: [0ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [3ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [3ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [11ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [12ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [13ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [13ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [31ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@1179364570(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@34a46fe2 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@191913761 after retry RPC@701213858), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@8455b86 -> wakeup thread Time-limited test -> (continuation of Deferred@191913761 after retry RPC@701213858))) after lookup
00:15:08.318 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] connecting to peer
00:15:08.319 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:15:08.319 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] cleaning up while in state CONNECTING due to: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:15:08.319 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:15:08.336 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:15:08.337 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367),70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)[L],b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)]}]
00:15:08.337 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=6, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=52), Traces: [0ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [3ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [3ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [11ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [12ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [13ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [13ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [31ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [33ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [34ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [34ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [47ms] refreshing cache from master, [47ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [51ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@1179364570(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@34a46fe2 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@191913761 after retry RPC@701213858) -> (continuation of Deferred@157171123 after retry RPC@2105814662), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@8455b86 -> wakeup thread Time-limited test -> (continuation of Deferred@191913761 after retry RPC@701213858) -> (continuation of Deferred@157171123 after retry RPC@2105814662))) after lookup
00:15:08.338 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] connecting to peer
00:15:08.339 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:15:08.339 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] cleaning up while in state CONNECTING due to: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:15:08.339 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:15:08.417 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:15:08.417 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367),70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)[L],b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)]}]
00:15:08.418 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=8, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=132), Traces: [0ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [3ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [3ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [11ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [12ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [13ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [13ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [31ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [33ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [34ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [34ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [47ms] refreshing cache from master, [47ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [51ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [52ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [54ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [54ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [127ms] refreshing cache from master, [127ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [132ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@1179364570(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@34a46fe2 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@191913761 after retry RPC@701213858) -> (continuation of Deferred@157171123 after retry RPC@2105814662) -> (continuation of Deferred@776221517 after retry RPC@1570515930), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@8455b86 -> wakeup thread Time-limited test -> (continuation of Deferred@191913761 after retry RPC@701213858) -> (continuation of Deferred@157171123 after retry RPC@2105814662) -> (continuation of Deferred@776221517 after retry RPC@1570515930))) after lookup
00:15:08.418 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] connecting to peer
00:15:08.420 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:15:08.420 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] cleaning up while in state CONNECTING due to: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:15:08.420 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:15:08.777 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:15:08.777 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367),70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)[L],b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)]}]
00:15:08.778 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=10, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=492), Traces: [0ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [3ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [3ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [11ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [12ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [13ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [13ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [31ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [33ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [34ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [34ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [47ms] refreshing cache from master, [47ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [51ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [52ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [54ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [54ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [127ms] refreshing cache from master, [127ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [132ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [133ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [135ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [135ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [487ms] refreshing cache from master, [487ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [491ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@1179364570(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@34a46fe2 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@191913761 after retry RPC@701213858) -> (continuation of Deferred@157171123 after retry RPC@2105814662) -> (continuation of Deferred@776221517 after retry RPC@1570515930) -> (continuation of Deferred@1776458577 after retry RPC@1436335948), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@8455b86 -> wakeup thread Time-limited test -> (continuation of Deferred@191913761 after retry RPC@701213858) -> (continuation of Deferred@157171123 after retry RPC@2105814662) -> (continuation of Deferred@776221517 after retry RPC@1570515930) -> (continuation of Deferred@1776458577 after retry RPC@1436335948))) after lookup
00:15:08.779 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] connecting to peer
00:15:08.780 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:15:08.780 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] cleaning up while in state CONNECTING due to: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:15:08.780 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:15:09.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:09.341392 32495 raft_consensus.cc:493] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
00:15:09.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:09.341892 32495 raft_consensus.cc:515] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:15:09.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:09.344352 32495 leader_election.cc:290] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b5d8d89741dd4568a4c7b210a4da72d2 (127.28.157.2:45951), 70a127fea97b464a89d03c5848cf6637 (127.28.157.1:37179)
00:15:09.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:09.350263 32272 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.157.1:37179: connect: Connection refused (error 111)
00:15:09.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:09.354678 32272 leader_election.cc:336] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 70a127fea97b464a89d03c5848cf6637 (127.28.157.1:37179): Network error: Client connection negotiation failed: client connection to 127.28.157.1:37179: connect: Connection refused (error 111)
00:15:09.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:09.357954 32068 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f4c1ce5aa4a4c8da912065e2decef6a" candidate_uuid: "094dbc44c6eb439083946b5ab0085ad0" candidate_term: 7 candidate_status { last_received { term: 6 index: 24 } } ignore_live_leader: false dest_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" is_pre_election: true
00:15:09.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:09.360335 32275 leader_election.cc:304] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 094dbc44c6eb439083946b5ab0085ad0; no voters: 70a127fea97b464a89d03c5848cf6637, b5d8d89741dd4568a4c7b210a4da72d2
00:15:09.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:09.361217 32495 raft_consensus.cc:2749] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
00:15:09.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:09.673179 32505 raft_consensus.cc:493] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 70a127fea97b464a89d03c5848cf6637)
00:15:09.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:09.673888 32505 raft_consensus.cc:515] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:15:09.677 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:15:09.677 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367),70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)[L],b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)]}]
00:15:09.677 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=12, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=1392), Traces: [0ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [3ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [3ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [11ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [12ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [13ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [13ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [31ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [33ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [34ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [34ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [47ms] refreshing cache from master, [47ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [51ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [52ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [54ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [54ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [127ms] refreshing cache from master, [127ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [132ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [133ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [135ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [135ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [487ms] refreshing cache from master, [487ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [491ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [493ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [495ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [495ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [1387ms] refreshing cache from master, [1387ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [1391ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@1179364570(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@34a46fe2 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@191913761 after retry RPC@701213858) -> (continuation of Deferred@157171123 after retry RPC@2105814662) -> (continuation of Deferred@776221517 after retry RPC@1570515930) -> (continuation of Deferred@1776458577 after retry RPC@1436335948) -> (continuation of Deferred@3098556 after retry RPC@821689898), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@8455b86 -> wakeup thread Time-limited test -> (continuation of Deferred@191913761 after retry RPC@701213858) -> (continuation of Deferred@157171123 after retry RPC@2105814662) -> (continuation of Deferred@776221517 after retry RPC@1570515930) -> (continuation of Deferred@1776458577 after retry RPC@1436335948) -> (continuation of Deferred@3098556 after retry RPC@821689898))) after lookup
00:15:09.678 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] connecting to peer
00:15:09.679 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:15:09.680 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] cleaning up while in state CONNECTING due to: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:15:09.680 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:15:09.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:09.679529 32505 leader_election.cc:290] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 094dbc44c6eb439083946b5ab0085ad0 (127.28.157.3:42367), 70a127fea97b464a89d03c5848cf6637 (127.28.157.1:37179)
00:15:09.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:09.693646 31947 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.157.1:37179: connect: Connection refused (error 111)
00:15:09.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:09.698498 31947 leader_election.cc:336] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 70a127fea97b464a89d03c5848cf6637 (127.28.157.1:37179): Network error: Client connection negotiation failed: client connection to 127.28.157.1:37179: connect: Connection refused (error 111)
00:15:09.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:09.702205 32392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f4c1ce5aa4a4c8da912065e2decef6a" candidate_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" candidate_term: 8 candidate_status { last_received { term: 7 index: 28 } } ignore_live_leader: false dest_uuid: "094dbc44c6eb439083946b5ab0085ad0" is_pre_election: true
00:15:09.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:09.702921 32392 raft_consensus.cc:2468] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5d8d89741dd4568a4c7b210a4da72d2 in term 6.
00:15:09.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:09.705072 31949 leader_election.cc:304] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 094dbc44c6eb439083946b5ab0085ad0, b5d8d89741dd4568a4c7b210a4da72d2; no voters: 70a127fea97b464a89d03c5848cf6637
00:15:09.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:09.706041 32505 raft_consensus.cc:2804] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [term 7 FOLLOWER]: Leader pre-election won for term 8
00:15:09.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:09.706380 32505 raft_consensus.cc:493] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 70a127fea97b464a89d03c5848cf6637)
00:15:09.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:09.706703 32505 raft_consensus.cc:3060] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [term 7 FOLLOWER]: Advancing to term 8
00:15:09.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:09.711726 32505 raft_consensus.cc:515] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:15:09.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:09.713336 32505 leader_election.cc:290] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [CANDIDATE]: Term 8 election: Requested vote from peers 094dbc44c6eb439083946b5ab0085ad0 (127.28.157.3:42367), 70a127fea97b464a89d03c5848cf6637 (127.28.157.1:37179)
00:15:09.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:09.714390 32392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f4c1ce5aa4a4c8da912065e2decef6a" candidate_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" candidate_term: 8 candidate_status { last_received { term: 7 index: 28 } } ignore_live_leader: false dest_uuid: "094dbc44c6eb439083946b5ab0085ad0"
00:15:09.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:09.714989 32392 raft_consensus.cc:3060] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 6 FOLLOWER]: Advancing to term 8
00:15:09.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:09.717161 31947 leader_election.cc:336] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 70a127fea97b464a89d03c5848cf6637 (127.28.157.1:37179): Network error: Client connection negotiation failed: client connection to 127.28.157.1:37179: connect: Connection refused (error 111)
00:15:09.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:09.722587 32392 raft_consensus.cc:2468] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5d8d89741dd4568a4c7b210a4da72d2 in term 8.
00:15:09.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:09.723574 31949 leader_election.cc:304] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 094dbc44c6eb439083946b5ab0085ad0, b5d8d89741dd4568a4c7b210a4da72d2; no voters: 70a127fea97b464a89d03c5848cf6637
00:15:09.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:09.724372 32505 raft_consensus.cc:2804] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [term 8 FOLLOWER]: Leader election won for term 8
00:15:09.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:09.726492 32505 raft_consensus.cc:697] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [term 8 LEADER]: Becoming Leader. State: Replica: b5d8d89741dd4568a4c7b210a4da72d2, State: Running, Role: LEADER
00:15:09.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:09.728080 32505 consensus_queue.cc:237] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 7.28, Last appended by leader: 28, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:15:09.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:09.738503 29607 catalog_manager.cc:5719] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 reported cstate change: term changed from 7 to 8, leader changed from 70a127fea97b464a89d03c5848cf6637 (127.28.157.1) to b5d8d89741dd4568a4c7b210a4da72d2 (127.28.157.2). New cstate: current_term: 8 leader_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } health_report { overall_health: UNKNOWN } } }
00:15:10.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:10.215308 32392 raft_consensus.cc:1275] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 8 FOLLOWER]: Refusing update from remote peer b5d8d89741dd4568a4c7b210a4da72d2: Log matching property violated. Preceding OpId in replica: term: 6 index: 24. Preceding OpId from leader: term: 8 index: 29. (index mismatch)
00:15:10.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:10.217340 32505 consensus_queue.cc:1048] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 24, Time since last communication: 0.001s
00:15:10.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:10.270466 31947 consensus_peers.cc:597] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 -> Peer 70a127fea97b464a89d03c5848cf6637 (127.28.157.1:37179): Couldn't send request to peer 70a127fea97b464a89d03c5848cf6637. Status: Network error: Client connection negotiation failed: client connection to 127.28.157.1:37179: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
00:15:12.696 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:15:12.697 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367),70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179),b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)[L]]}]
00:15:12.697 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=14, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=4412), Traces: [0ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [3ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [3ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [11ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [12ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [13ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [13ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [31ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [33ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [34ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [34ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [47ms] refreshing cache from master, [47ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [51ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [52ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [54ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [54ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [127ms] refreshing cache from master, [127ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [132ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [133ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [135ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [135ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [487ms] refreshing cache from master, [487ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [491ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [493ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [495ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [495ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [1387ms] refreshing cache from master, [1387ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [1391ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [1393ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [1395ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [1395ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [4407ms] refreshing cache from master, [4407ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [4411ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@1179364570(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@34a46fe2 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@191913761 after retry RPC@701213858) -> (continuation of Deferred@157171123 after retry RPC@2105814662) -> (continuation of Deferred@776221517 after retry RPC@1570515930) -> (continuation of Deferred@1776458577 after retry RPC@1436335948) -> (continuation of Deferred@3098556 after retry RPC@821689898) -> (continuation of Deferred@2039525197 after retry RPC@115807995), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@8455b86 -> wakeup thread Time-limited test -> (continuation of Deferred@191913761 after retry RPC@701213858) -> (continuation of Deferred@157171123 after retry RPC@2105814662) -> (continuation of Deferred@776221517 after retry RPC@1570515930) -> (continuation of Deferred@1776458577 after retry RPC@1436335948) -> (continuation of Deferred@3098556 after retry RPC@821689898) -> (continuation of Deferred@2039525197 after retry RPC@115807995))) after lookup
00:15:12.698 [DEBUG - kudu-nio-2] (Connection.java:748) [peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)] connecting to peer
00:15:12.699 [DEBUG - kudu-nio-6] (Connection.java:757) [peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)] Successfully connected to peer
00:15:12.718 [DEBUG - kudu-nio-6] (Negotiator.java:918) Authenticated connection [id: 0x1853cb71, L:/127.0.0.1:58576 - R:/127.28.157.2:45951] using TOKEN/null
00:15:12.779 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.28.157.1:37179
00:15:12.779 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

00:15:12.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:12.781010 29300 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu
00:15:12.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu
00:15:12.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/wal
00:15:12.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/data
00:15:12.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
00:15:12.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
00:15:12.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
00:15:12.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
00:15:12.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
00:15:12.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
00:15:12.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/logs
00:15:12.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/data/info.pb
00:15:12.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
00:15:12.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
00:15:12.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
00:15:12.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
00:15:12.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
00:15:12.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
00:15:12.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
00:15:12.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
00:15:12.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
00:15:12.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
00:15:12.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.28.157.1:37179
00:15:12.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.28.157.1
00:15:12.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.28.157.62:35965,127.28.157.61:44801,127.28.157.60:33393
00:15:12.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=35519
00:15:12.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.28.157.1
00:15:12.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.28.157.20:34671
00:15:12.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
00:15:12.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
00:15:12.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
00:15:12.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:12.969362 31947 consensus_peers.cc:597] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 -> Peer 70a127fea97b464a89d03c5848cf6637 (127.28.157.1:37179): Couldn't send request to peer 70a127fea97b464a89d03c5848cf6637. Status: Network error: Client connection negotiation failed: client connection to 127.28.157.1:37179: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
00:15:13.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:13.131829 32581 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
00:15:13.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:13.132500 32581 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
00:15:13.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:13.132972 32581 flags.cc:432] Enabled unsafe flag: --never_fsync=true
00:15:13.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:13.167662 32581 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
00:15:13.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:13.168654 32581 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.157.1
00:15:13.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:13.207741 32581 tablet_server_runner.cc:78] Tablet server non-default flags:
00:15:13.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/logs
00:15:13.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
00:15:13.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
00:15:13.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
00:15:13.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.28.157.20:34671
00:15:13.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
00:15:13.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
00:15:13.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/data
00:15:13.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/wal
00:15:13.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
00:15:13.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
00:15:13.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.28.157.1:37179
00:15:13.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
00:15:13.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
00:15:13.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
00:15:13.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/data/info.pb
00:15:13.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.28.157.1
00:15:13.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=35519
00:15:13.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.28.157.62:35965,127.28.157.61:44801,127.28.157.60:33393
00:15:13.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
00:15:13.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
00:15:13.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
00:15:13.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
00:15:13.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
00:15:13.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.28.157.1
00:15:13.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
00:15:13.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
00:15:13.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
00:15:13.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision f7ed5b56c096f54e2f926bf94ceb7fc44c0c1329
00:15:13.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
00:15:13.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 11 Aug 2026 23:55:14 UTC on e7f111948823
00:15:13.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 17639
00:15:13.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
00:15:13.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:13.209525 32581 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
00:15:13.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:13.211302 32581 file_cache.cc:504] Constructed file cache file cache with capacity 419430
00:15:13.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:13.231900 32591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
00:15:14.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:13.234174 32592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
00:15:14.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:14.431425 32593 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
00:15:14.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:14.433817 32599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
00:15:14.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:14.436405 32581 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.203s	user 0.497s	sys 0.698s
00:15:14.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:14.436689 32581 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.203s	user 0.497s	sys 0.698s
00:15:14.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:14.436985 32581 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
00:15:14.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:14.438374 32581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
00:15:14.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:14.440949 32581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
00:15:14.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:14.442374 32581 hybrid_clock.cc:648] HybridClock initialized: now 1786493714442337 us; error 54 us; skew 500 ppm
00:15:14.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:14.443300 32581 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
00:15:14.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:14.451589 32581 webserver.cc:533] Webserver started at http://127.28.157.1:35519/ using document root <none> and password file <none>
00:15:14.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:14.452913 32581 fs_manager.cc:362] Metadata directory not provided
00:15:14.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:14.453154 32581 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
00:15:14.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:14.462888 32581 fs_manager.cc:714] Time spent opening directory manager: real 0.007s	user 0.003s	sys 0.004s
00:15:14.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:14.469108 32618 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
00:15:14.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:14.470623 32581 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.003s	sys 0.003s
00:15:14.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:14.471079 32581 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/data,/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/wal
00:15:14.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "70a127fea97b464a89d03c5848cf6637"
00:15:14.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 00:14:32 on dist-test-slave-t7g5"
00:15:14.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:14.474717 32581 fs_report.cc:389] FS layout report
00:15:14.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
00:15:14.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/wal
00:15:14.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/wal
00:15:14.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/data/data
00:15:14.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
00:15:14.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
00:15:14.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
00:15:14.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
00:15:14.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
00:15:14.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
00:15:14.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
00:15:14.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
00:15:14.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
00:15:14.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
00:15:14.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
00:15:14.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:14.581238 32581 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
00:15:14.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:14.582854 32581 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
00:15:14.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:14.583269 32581 kserver.cc:163] Server-wide thread pool size limit: 3276
00:15:14.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:14.586086 32581 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
00:15:14.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:14.592768 32629 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
00:15:14.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:14.605167 32581 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
00:15:14.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:14.605450 32581 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.014s	user 0.001s	sys 0.001s
00:15:14.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:14.605784 32581 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
00:15:14.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:14.610904 32581 ts_tablet_manager.cc:616] Registered 1 tablets
00:15:14.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:14.611114 32581 ts_tablet_manager.cc:595] Time spent register tablets: real 0.005s	user 0.004s	sys 0.000s
00:15:14.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:14.611686 32629 tablet_bootstrap.cc:492] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637: Bootstrap starting.
00:15:14.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:14.702461 32629 log.cc:826] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637: Log is configured to *not* fsync() on all Append() calls
00:15:14.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:14.787976 31947 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.157.1:37179: connect: Connection refused (error 111) [suppressed 10 similar messages]
00:15:14.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:14.814802 32629 tablet_bootstrap.cc:492] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=21 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
00:15:14.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:14.815900 32629 tablet_bootstrap.cc:492] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637: Bootstrap complete.
00:15:14.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:14.817800 32629 ts_tablet_manager.cc:1403] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637: Time spent bootstrapping tablet: real 0.207s	user 0.160s	sys 0.034s
00:15:14.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:14.843117 32629 raft_consensus.cc:359] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [term 7 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:15:14.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:14.846441 32629 raft_consensus.cc:740] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70a127fea97b464a89d03c5848cf6637, State: Initialized, Role: FOLLOWER
00:15:14.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:14.847918 32629 consensus_queue.cc:260] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 7.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:15:14.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:14.851984 32629 ts_tablet_manager.cc:1434] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637: Time spent starting tablet: real 0.034s	user 0.029s	sys 0.003s
00:15:14.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:14.911777 32581 rpc_server.cc:307] RPC server started. Bound to: 127.28.157.1:37179
00:15:14.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:14.912106   363 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.157.1:37179 every 8 connection(s)
00:15:14.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:14.914548 32581 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/data/info.pb
00:15:14.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:14.916716 29300 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu as pid 32581
00:15:14.917 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
00:15:14.980 [DEBUG - kudu-nio-6] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 24 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
00:15:14.981 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.28.157.2:45951
00:15:14.981 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

00:15:14.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:14.982800 29300 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu with pid 31907
00:15:15.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:15.007076   366 heartbeater.cc:344] Connected to a master server at 127.28.157.61:44801
00:15:15.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:15.007668   366 heartbeater.cc:461] Registering TS with master...
00:15:15.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:15.008873   365 heartbeater.cc:344] Connected to a master server at 127.28.157.60:33393
00:15:15.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:15.009109   366 heartbeater.cc:507] Master 127.28.157.61:44801 requested a full tablet report, sending...
00:15:15.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:15.009245   365 heartbeater.cc:461] Registering TS with master...
00:15:15.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:15.010270   365 heartbeater.cc:507] Master 127.28.157.60:33393 requested a full tablet report, sending...
00:15:15.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:15.014081   367 heartbeater.cc:344] Connected to a master server at 127.28.157.62:35965
00:15:15.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:15.014463   367 heartbeater.cc:461] Registering TS with master...
00:15:15.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:15.015026 29481 ts_manager.cc:194] Re-registered known tserver with Master: 70a127fea97b464a89d03c5848cf6637 (127.28.157.1:37179)
00:15:15.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:15.015244 29607 ts_manager.cc:194] Re-registered known tserver with Master: 70a127fea97b464a89d03c5848cf6637 (127.28.157.1:37179)
00:15:15.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:15.015404   367 heartbeater.cc:507] Master 127.28.157.62:35965 requested a full tablet report, sending...
00:15:15.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:15.019259 29607 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.28.157.1:40233
00:15:15.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:15.019212 29360 ts_manager.cc:194] Re-registered known tserver with Master: 70a127fea97b464a89d03c5848cf6637 (127.28.157.1:37179)
00:15:15.047 [DEBUG - kudu-nio-6] (Connection.java:239) [peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)] handling channelInactive
00:15:15.047 [DEBUG - kudu-nio-6] (Connection.java:709) [peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)] cleaning up while in state READY due to: connection closed
00:15:15.048 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
00:15:15.048 [DEBUG - Time-limited test] (Connection.java:748) [peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)] connecting to peer
00:15:15.050 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951
00:15:15.050 [DEBUG - kudu-nio-7] (Connection.java:709) [peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)] cleaning up while in state CONNECTING due to: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951
00:15:15.050 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951
00:15:15.057 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:15:15.057 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367),70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179),b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)[L]]}]
00:15:15.057 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=2, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=9), Traces: [0ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [2ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [2ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [9ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@604337954(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@1aebfd97 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4c09033d -> wakeup thread Time-limited test)) after lookup
00:15:15.058 [DEBUG - kudu-nio-2] (Connection.java:748) [peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)] connecting to peer
00:15:15.059 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951
00:15:15.060 [DEBUG - kudu-nio-0] (Connection.java:709) [peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)] cleaning up while in state CONNECTING due to: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951
00:15:15.060 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951
00:15:15.076 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:15:15.077 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367),70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179),b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)[L]]}]
00:15:15.077 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=4, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=29), Traces: [0ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [2ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [2ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [9ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [10ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [12ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [12ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [24ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [28ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@604337954(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@1aebfd97 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1151634918 after retry RPC@1085304484), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4c09033d -> wakeup thread Time-limited test -> (continuation of Deferred@1151634918 after retry RPC@1085304484))) after lookup
00:15:15.078 [DEBUG - kudu-nio-2] (Connection.java:748) [peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)] connecting to peer
00:15:15.079 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951
00:15:15.079 [DEBUG - kudu-nio-1] (Connection.java:709) [peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)] cleaning up while in state CONNECTING due to: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951
00:15:15.079 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951
00:15:15.096 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:15:15.097 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367),70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179),b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)[L]]}]
00:15:15.097 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=6, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=49), Traces: [0ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [2ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [2ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [9ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [10ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [12ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [12ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [24ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [28ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [29ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [31ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [31ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [44ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [48ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@604337954(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@1aebfd97 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1151634918 after retry RPC@1085304484) -> (continuation of Deferred@829281185 after retry RPC@1928027750), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4c09033d -> wakeup thread Time-limited test -> (continuation of Deferred@1151634918 after retry RPC@1085304484) -> (continuation of Deferred@829281185 after retry RPC@1928027750))) after lookup
00:15:15.098 [DEBUG - kudu-nio-2] (Connection.java:748) [peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)] connecting to peer
00:15:15.099 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951
00:15:15.099 [DEBUG - kudu-nio-2] (Connection.java:709) [peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)] cleaning up while in state CONNECTING due to: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951
00:15:15.099 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951
00:15:15.116 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:15:15.116 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367),70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179),b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)[L]]}]
00:15:15.117 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=8, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=68), Traces: [0ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [2ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [2ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [9ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [10ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [12ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [12ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [24ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [28ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [29ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [31ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [31ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [44ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [48ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [49ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [51ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [51ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [68ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@604337954(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@1aebfd97 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1151634918 after retry RPC@1085304484) -> (continuation of Deferred@829281185 after retry RPC@1928027750) -> (continuation of Deferred@1830428243 after retry RPC@2126668614), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4c09033d -> wakeup thread Time-limited test -> (continuation of Deferred@1151634918 after retry RPC@1085304484) -> (continuation of Deferred@829281185 after retry RPC@1928027750) -> (continuation of Deferred@1830428243 after retry RPC@2126668614))) after lookup
00:15:15.117 [DEBUG - kudu-nio-2] (Connection.java:748) [peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)] connecting to peer
00:15:15.118 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951
00:15:15.119 [DEBUG - kudu-nio-3] (Connection.java:709) [peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)] cleaning up while in state CONNECTING due to: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951
00:15:15.119 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951
00:15:15.336 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:15:15.337 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367),70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179),b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)[L]]}]
00:15:15.337 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=10, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=289), Traces: [0ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [2ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [2ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [9ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [10ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [12ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [12ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [24ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [28ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [29ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [31ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [31ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [44ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [48ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [49ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [51ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [51ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [68ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [69ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [71ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [71ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [284ms] refreshing cache from master, [284ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [288ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@604337954(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@1aebfd97 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1151634918 after retry RPC@1085304484) -> (continuation of Deferred@829281185 after retry RPC@1928027750) -> (continuation of Deferred@1830428243 after retry RPC@2126668614) -> (continuation of Deferred@1760166679 after retry RPC@98534687), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4c09033d -> wakeup thread Time-limited test -> (continuation of Deferred@1151634918 after retry RPC@1085304484) -> (continuation of Deferred@829281185 after retry RPC@1928027750) -> (continuation of Deferred@1830428243 after retry RPC@2126668614) -> (continuation of Deferred@1760166679 after retry RPC@98534687))) after lookup
00:15:15.338 [DEBUG - kudu-nio-2] (Connection.java:748) [peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)] connecting to peer
00:15:15.340 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951
00:15:15.340 [DEBUG - kudu-nio-4] (Connection.java:709) [peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)] cleaning up while in state CONNECTING due to: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951
00:15:15.340 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951
00:15:15.876 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:15:15.876 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367),70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179),b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)[L]]}]
00:15:15.877 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=12, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=828), Traces: [0ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [2ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [2ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [9ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [10ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [12ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [12ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [24ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [28ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [29ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [31ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [31ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [44ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [48ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [49ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [51ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [51ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [68ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [69ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [71ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [71ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [284ms] refreshing cache from master, [284ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [288ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [290ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [292ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [292ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [824ms] refreshing cache from master, [824ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [828ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@604337954(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@1aebfd97 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1151634918 after retry RPC@1085304484) -> (continuation of Deferred@829281185 after retry RPC@1928027750) -> (continuation of Deferred@1830428243 after retry RPC@2126668614) -> (continuation of Deferred@1760166679 after retry RPC@98534687) -> (continuation of Deferred@1390793326 after retry RPC@728449616), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4c09033d -> wakeup thread Time-limited test -> (continuation of Deferred@1151634918 after retry RPC@1085304484) -> (continuation of Deferred@829281185 after retry RPC@1928027750) -> (continuation of Deferred@1830428243 after retry RPC@2126668614) -> (continuation of Deferred@1760166679 after retry RPC@98534687) -> (continuation of Deferred@1390793326 after retry RPC@728449616))) after lookup
00:15:15.877 [DEBUG - kudu-nio-2] (Connection.java:748) [peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)] connecting to peer
00:15:15.879 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951
00:15:15.879 [DEBUG - kudu-nio-5] (Connection.java:709) [peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)] cleaning up while in state CONNECTING due to: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951
00:15:15.879 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951
00:15:16.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:16.023695   365 heartbeater.cc:499] Master 127.28.157.60:33393 was elected leader, sending a full tablet report...
00:15:16.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:16.294128   382 raft_consensus.cc:493] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b5d8d89741dd4568a4c7b210a4da72d2)
00:15:16.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:16.295195   382 raft_consensus.cc:515] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:15:16.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:16.297477   382 leader_election.cc:290] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b5d8d89741dd4568a4c7b210a4da72d2 (127.28.157.2:45951), 70a127fea97b464a89d03c5848cf6637 (127.28.157.1:37179)
00:15:16.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:16.314656 32275 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.157.2:45951: connect: Connection refused (error 111)
00:15:16.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:16.315065   387 raft_consensus.cc:493] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
00:15:16.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:16.315680   387 raft_consensus.cc:515] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:15:16.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:16.319305 32275 leader_election.cc:336] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer b5d8d89741dd4568a4c7b210a4da72d2 (127.28.157.2:45951): Network error: Client connection negotiation failed: client connection to 127.28.157.2:45951: connect: Connection refused (error 111)
00:15:16.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:16.321570   387 leader_election.cc:290] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b5d8d89741dd4568a4c7b210a4da72d2 (127.28.157.2:45951), 094dbc44c6eb439083946b5ab0085ad0 (127.28.157.3:42367)
00:15:16.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:16.332531 32625 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.157.2:45951: connect: Connection refused (error 111)
00:15:16.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:16.334786 32751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f4c1ce5aa4a4c8da912065e2decef6a" candidate_uuid: "094dbc44c6eb439083946b5ab0085ad0" candidate_term: 9 candidate_status { last_received { term: 8 index: 32 } } ignore_live_leader: false dest_uuid: "70a127fea97b464a89d03c5848cf6637" is_pre_election: true
00:15:16.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:16.335711 32751 raft_consensus.cc:2468] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 094dbc44c6eb439083946b5ab0085ad0 in term 7.
00:15:16.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:16.337492 32272 leader_election.cc:304] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 094dbc44c6eb439083946b5ab0085ad0, 70a127fea97b464a89d03c5848cf6637; no voters: b5d8d89741dd4568a4c7b210a4da72d2
00:15:16.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:16.338500   382 raft_consensus.cc:2804] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 8 FOLLOWER]: Leader pre-election won for term 9
00:15:16.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:16.338865   382 raft_consensus.cc:493] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 8 FOLLOWER]: Starting leader election (detected failure of leader b5d8d89741dd4568a4c7b210a4da72d2)
00:15:16.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:16.338559 32625 leader_election.cc:336] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer b5d8d89741dd4568a4c7b210a4da72d2 (127.28.157.2:45951): Network error: Client connection negotiation failed: client connection to 127.28.157.2:45951: connect: Connection refused (error 111)
00:15:16.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:16.339115   382 raft_consensus.cc:3060] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 8 FOLLOWER]: Advancing to term 9
00:15:16.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:16.343537 32392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f4c1ce5aa4a4c8da912065e2decef6a" candidate_uuid: "70a127fea97b464a89d03c5848cf6637" candidate_term: 8 candidate_status { last_received { term: 7 index: 28 } } ignore_live_leader: false dest_uuid: "094dbc44c6eb439083946b5ab0085ad0" is_pre_election: true
00:15:16.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:16.344826   382 raft_consensus.cc:515] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:15:16.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:16.346124 32392 raft_consensus.cc:2368] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 70a127fea97b464a89d03c5848cf6637 for earlier term 8. Current term is 9.
00:15:16.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:16.346558   382 leader_election.cc:290] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [CANDIDATE]: Term 9 election: Requested vote from peers b5d8d89741dd4568a4c7b210a4da72d2 (127.28.157.2:45951), 70a127fea97b464a89d03c5848cf6637 (127.28.157.1:37179)
00:15:16.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:16.348268 32751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f4c1ce5aa4a4c8da912065e2decef6a" candidate_uuid: "094dbc44c6eb439083946b5ab0085ad0" candidate_term: 9 candidate_status { last_received { term: 8 index: 32 } } ignore_live_leader: false dest_uuid: "70a127fea97b464a89d03c5848cf6637"
00:15:16.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:16.348891 32751 raft_consensus.cc:3060] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [term 7 FOLLOWER]: Advancing to term 9
00:15:16.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:16.352541 32275 leader_election.cc:336] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer b5d8d89741dd4568a4c7b210a4da72d2 (127.28.157.2:45951): Network error: Client connection negotiation failed: client connection to 127.28.157.2:45951: connect: Connection refused (error 111)
00:15:16.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:16.352847 32622 leader_election.cc:400] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [CANDIDATE]: Term 8 pre-election: Vote denied by peer 094dbc44c6eb439083946b5ab0085ad0 (127.28.157.3:42367) with higher term. Message: Invalid argument: T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 70a127fea97b464a89d03c5848cf6637 for earlier term 8. Current term is 9.
00:15:16.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:16.353154 32622 leader_election.cc:403] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [CANDIDATE]: Term 8 pre-election: Cancelling election due to peer responding with higher term
00:15:16.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:16.357746 32751 raft_consensus.cc:2468] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 094dbc44c6eb439083946b5ab0085ad0 in term 9.
00:15:16.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:16.358382   387 raft_consensus.cc:2749] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: Vote denied by peer 094dbc44c6eb439083946b5ab0085ad0 (127.28.157.3:42367) with higher term. Message: Invalid argument: T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 70a127fea97b464a89d03c5848cf6637 for earlier term 8. Current term is 9.
00:15:16.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:16.359282 32272 leader_election.cc:304] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 094dbc44c6eb439083946b5ab0085ad0, 70a127fea97b464a89d03c5848cf6637; no voters: b5d8d89741dd4568a4c7b210a4da72d2
00:15:16.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:16.360447   382 raft_consensus.cc:2804] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 9 FOLLOWER]: Leader election won for term 9
00:15:16.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:16.361956   382 raft_consensus.cc:697] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 9 LEADER]: Becoming Leader. State: Replica: 094dbc44c6eb439083946b5ab0085ad0, State: Running, Role: LEADER
00:15:16.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:16.363276   382 consensus_queue.cc:237] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 8.32, Last appended by leader: 32, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:15:16.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:16.374162 29607 catalog_manager.cc:5719] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 reported cstate change: term changed from 8 to 9, leader changed from b5d8d89741dd4568a4c7b210a4da72d2 (127.28.157.2) to 094dbc44c6eb439083946b5ab0085ad0 (127.28.157.3). New cstate: current_term: 9 leader_uuid: "094dbc44c6eb439083946b5ab0085ad0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } health_report { overall_health: UNKNOWN } } }
00:15:16.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:16.808285 32751 raft_consensus.cc:1275] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [term 9 FOLLOWER]: Refusing update from remote peer 094dbc44c6eb439083946b5ab0085ad0: Log matching property violated. Preceding OpId in replica: term: 7 index: 28. Preceding OpId from leader: term: 9 index: 33. (index mismatch)
00:15:16.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:16.810812   382 consensus_queue.cc:1048] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 28, Time since last communication: 0.000s
00:15:16.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:16.853318 32275 consensus_peers.cc:597] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 -> Peer b5d8d89741dd4568a4c7b210a4da72d2 (127.28.157.2:45951): Couldn't send request to peer b5d8d89741dd4568a4c7b210a4da72d2. Status: Network error: Client connection negotiation failed: client connection to 127.28.157.2:45951: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
00:15:17.036 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:15:17.037 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)[L],70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179),b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)]}]
00:15:17.038 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=14, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=1989), Traces: [0ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [2ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [2ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [9ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [10ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [12ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [12ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [24ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [28ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [29ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [31ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [31ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [44ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [48ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [49ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [51ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [51ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [68ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [69ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [71ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [71ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [284ms] refreshing cache from master, [284ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [288ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [290ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [292ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [292ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [824ms] refreshing cache from master, [824ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [828ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [829ms] sending RPC to server b5d8d89741dd4568a4c7b210a4da72d2, [831ms] received response from server b5d8d89741dd4568a4c7b210a4da72d2: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [831ms] delaying RPC due to: Network error: Failed to connect to peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951): Connection refused: /127.28.157.2:45951, [1984ms] refreshing cache from master, [1984ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [1988ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@604337954(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@1aebfd97 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1151634918 after retry RPC@1085304484) -> (continuation of Deferred@829281185 after retry RPC@1928027750) -> (continuation of Deferred@1830428243 after retry RPC@2126668614) -> (continuation of Deferred@1760166679 after retry RPC@98534687) -> (continuation of Deferred@1390793326 after retry RPC@728449616) -> (continuation of Deferred@732607753 after retry RPC@1512184814), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4c09033d -> wakeup thread Time-limited test -> (continuation of Deferred@1151634918 after retry RPC@1085304484) -> (continuation of Deferred@829281185 after retry RPC@1928027750) -> (continuation of Deferred@1830428243 after retry RPC@2126668614) -> (continuation of Deferred@1760166679 after retry RPC@98534687) -> (continuation of Deferred@1390793326 after retry RPC@728449616) -> (continuation of Deferred@732607753 after retry RPC@1512184814))) after lookup
00:15:17.039 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)] connecting to peer
00:15:17.040 [DEBUG - kudu-nio-6] (Connection.java:757) [peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)] Successfully connected to peer
00:15:17.065 [DEBUG - kudu-nio-6] (Negotiator.java:918) Authenticated connection [id: 0x33fe0719, L:/127.0.0.1:50696 - R:/127.28.157.3:42367] using TOKEN/null
00:15:17.122 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.28.157.2:45951
00:15:17.123 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

00:15:17.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:17.124838 29300 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu
00:15:17.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu
00:15:17.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-1/wal
00:15:17.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-1/data
00:15:17.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
00:15:17.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
00:15:17.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
00:15:17.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
00:15:17.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
00:15:17.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
00:15:17.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-1/logs
00:15:17.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-1/data/info.pb
00:15:17.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
00:15:17.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
00:15:17.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
00:15:17.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
00:15:17.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
00:15:17.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
00:15:17.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
00:15:17.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
00:15:17.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
00:15:17.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
00:15:17.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.28.157.2:45951
00:15:17.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.28.157.2
00:15:17.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.28.157.62:35965,127.28.157.61:44801,127.28.157.60:33393
00:15:17.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40801
00:15:17.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.28.157.2
00:15:17.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.28.157.20:34671
00:15:17.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
00:15:17.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
00:15:17.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
00:15:17.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:17.502950   434 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
00:15:17.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:17.503477   434 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
00:15:17.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:17.504046   434 flags.cc:432] Enabled unsafe flag: --never_fsync=true
00:15:17.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:17.540735   434 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
00:15:17.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:17.541635   434 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.157.2
00:15:17.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:17.581763   434 tablet_server_runner.cc:78] Tablet server non-default flags:
00:15:17.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-1/logs
00:15:17.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
00:15:17.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
00:15:17.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
00:15:17.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.28.157.20:34671
00:15:17.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
00:15:17.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
00:15:17.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-1/data
00:15:17.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-1/wal
00:15:17.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
00:15:17.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
00:15:17.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.28.157.2:45951
00:15:17.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
00:15:17.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
00:15:17.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
00:15:17.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-1/data/info.pb
00:15:17.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.28.157.2
00:15:17.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40801
00:15:17.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.28.157.62:35965,127.28.157.61:44801,127.28.157.60:33393
00:15:17.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
00:15:17.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
00:15:17.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
00:15:17.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
00:15:17.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
00:15:17.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.28.157.2
00:15:17.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
00:15:17.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
00:15:17.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
00:15:17.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision f7ed5b56c096f54e2f926bf94ceb7fc44c0c1329
00:15:17.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
00:15:17.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 11 Aug 2026 23:55:14 UTC on e7f111948823
00:15:17.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 17639
00:15:17.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
00:15:17.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:17.583197   434 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
00:15:17.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:17.585059   434 file_cache.cc:504] Constructed file cache file cache with capacity 419430
00:15:17.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:17.600687   447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
00:15:18.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:17.601078   448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
00:15:18.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:17.605356   450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
00:15:18.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:18.772406   449 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
00:15:18.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:18.772838   434 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
00:15:18.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:18.778250   434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
00:15:18.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:18.781071   434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
00:15:18.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:18.782783   434 hybrid_clock.cc:648] HybridClock initialized: now 1786493718782736 us; error 65 us; skew 500 ppm
00:15:18.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:18.783902   434 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
00:15:18.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:18.791637   434 webserver.cc:533] Webserver started at http://127.28.157.2:40801/ using document root <none> and password file <none>
00:15:18.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:18.792887   434 fs_manager.cc:362] Metadata directory not provided
00:15:18.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:18.793154   434 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
00:15:18.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:18.803208   434 fs_manager.cc:714] Time spent opening directory manager: real 0.007s	user 0.006s	sys 0.001s
00:15:18.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:18.811398   463 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
00:15:18.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:18.812856   434 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.003s	sys 0.003s
00:15:18.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:18.813280   434 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-1/data,/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-1/wal
00:15:18.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "b5d8d89741dd4568a4c7b210a4da72d2"
00:15:18.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 00:14:34 on dist-test-slave-t7g5"
00:15:18.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:18.815898   434 fs_report.cc:389] FS layout report
00:15:18.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
00:15:18.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-1/wal
00:15:18.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-1/wal
00:15:18.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-1/data/data
00:15:18.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
00:15:18.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
00:15:18.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
00:15:18.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
00:15:18.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
00:15:18.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
00:15:18.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
00:15:18.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
00:15:18.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
00:15:18.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
00:15:18.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
00:15:18.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:18.885251   434 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
00:15:18.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:18.887614   434 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
00:15:18.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:18.888236   434 kserver.cc:163] Server-wide thread pool size limit: 3276
00:15:18.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:18.892810   434 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
00:15:18.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:18.903293   478 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
00:15:18.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:18.916287   434 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
00:15:18.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:18.916563   434 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.016s	user 0.000s	sys 0.003s
00:15:18.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:18.916819   434 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
00:15:18.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:18.922645   434 ts_tablet_manager.cc:616] Registered 1 tablets
00:15:18.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:18.923015   434 ts_tablet_manager.cc:595] Time spent register tablets: real 0.006s	user 0.005s	sys 0.000s
00:15:18.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:18.923640   478 tablet_bootstrap.cc:492] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2: Bootstrap starting.
00:15:19.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:19.007740   478 log.cc:826] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2: Log is configured to *not* fsync() on all Append() calls
00:15:19.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:19.138688   434 rpc_server.cc:307] RPC server started. Bound to: 127.28.157.2:45951
00:15:19.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:19.138851   651 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.157.2:45951 every 8 connection(s)
00:15:19.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:19.141996   434 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-1/data/info.pb
00:15:19.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:19.150843 29300 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu as pid 434
00:15:19.151 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
00:15:19.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:19.173285   478 tablet_bootstrap.cc:492] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=24 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
00:15:19.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:19.176342   478 tablet_bootstrap.cc:492] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2: Bootstrap complete.
00:15:19.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:19.196413   478 ts_tablet_manager.cc:1403] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2: Time spent bootstrapping tablet: real 0.273s	user 0.183s	sys 0.048s
00:15:19.215 [DEBUG - kudu-nio-6] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 27 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
00:15:19.217 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.28.157.3:42367
00:15:19.218 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

00:15:19.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:19.218638 29300 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu with pid 32236
00:15:19.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:19.238765   652 heartbeater.cc:344] Connected to a master server at 127.28.157.60:33393
00:15:19.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:19.239425   652 heartbeater.cc:461] Registering TS with master...
00:15:19.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:19.241128   652 heartbeater.cc:507] Master 127.28.157.60:33393 requested a full tablet report, sending...
00:15:19.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:19.246949   656 heartbeater.cc:344] Connected to a master server at 127.28.157.62:35965
00:15:19.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:19.247285   656 heartbeater.cc:461] Registering TS with master...
00:15:19.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:19.248303   656 heartbeater.cc:507] Master 127.28.157.62:35965 requested a full tablet report, sending...
00:15:19.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:19.248520 29607 ts_manager.cc:194] Re-registered known tserver with Master: b5d8d89741dd4568a4c7b210a4da72d2 (127.28.157.2:45951)
00:15:19.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:19.252964 29607 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.28.157.2:44219
00:15:19.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:19.252964 29361 ts_manager.cc:194] Re-registered known tserver with Master: b5d8d89741dd4568a4c7b210a4da72d2 (127.28.157.2:45951)
00:15:19.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:19.259754   655 heartbeater.cc:344] Connected to a master server at 127.28.157.61:44801
00:15:19.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:19.260232   655 heartbeater.cc:461] Registering TS with master...
00:15:19.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:19.261190   655 heartbeater.cc:507] Master 127.28.157.61:44801 requested a full tablet report, sending...
00:15:19.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:19.263805   478 raft_consensus.cc:359] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [term 8 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:15:19.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:19.266320   478 raft_consensus.cc:740] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5d8d89741dd4568a4c7b210a4da72d2, State: Initialized, Role: FOLLOWER
00:15:19.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:19.267416   478 consensus_queue.cc:260] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 8.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:15:19.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:19.268736 29481 ts_manager.cc:194] Re-registered known tserver with Master: b5d8d89741dd4568a4c7b210a4da72d2 (127.28.157.2:45951)
00:15:19.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:19.272159   652 heartbeater.cc:499] Master 127.28.157.60:33393 was elected leader, sending a full tablet report...
00:15:19.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:19.272894   478 ts_tablet_manager.cc:1434] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2: Time spent starting tablet: real 0.076s	user 0.028s	sys 0.001s
00:15:19.287 [DEBUG - kudu-nio-6] (Connection.java:239) [peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)] handling channelInactive
00:15:19.287 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)] cleaning up while in state READY due to: connection closed
00:15:19.288 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
00:15:19.289 [DEBUG - Time-limited test] (Connection.java:748) [peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)] connecting to peer
00:15:19.290 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367
00:15:19.290 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)] cleaning up while in state CONNECTING due to: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367
00:15:19.290 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367
00:15:19.297 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:15:19.297 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)[L],70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179),b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)]}]
00:15:19.297 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=2, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=9), Traces: [0ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [2ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [2ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [9ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@1598053642(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@15f7c067 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6d8fb89a -> wakeup thread Time-limited test)) after lookup
00:15:19.298 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)] connecting to peer
00:15:19.299 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367
00:15:19.299 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)] cleaning up while in state CONNECTING due to: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367
00:15:19.300 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367
00:15:19.318 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:15:19.318 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)[L],70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179),b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)]}]
00:15:19.318 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=4, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=30), Traces: [0ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [2ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [2ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [9ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [10ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [12ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [12ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [24ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [29ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@1598053642(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@15f7c067 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1600906042 after retry RPC@1328601893), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6d8fb89a -> wakeup thread Time-limited test -> (continuation of Deferred@1600906042 after retry RPC@1328601893))) after lookup
00:15:19.319 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)] connecting to peer
00:15:19.320 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367
00:15:19.320 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)] cleaning up while in state CONNECTING due to: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367
00:15:19.320 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367
00:15:19.359 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:15:19.359 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)[L],70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179),b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)]}]
00:15:19.359 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=6, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=71), Traces: [0ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [2ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [2ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [9ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [10ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [12ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [12ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [24ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [29ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [30ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [32ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [32ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [64ms] refreshing cache from master, [65ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [70ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@1598053642(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@15f7c067 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1600906042 after retry RPC@1328601893) -> (continuation of Deferred@1069954512 after retry RPC@1986410750), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6d8fb89a -> wakeup thread Time-limited test -> (continuation of Deferred@1600906042 after retry RPC@1328601893) -> (continuation of Deferred@1069954512 after retry RPC@1986410750))) after lookup
00:15:19.360 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)] connecting to peer
00:15:19.362 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367
00:15:19.362 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)] cleaning up while in state CONNECTING due to: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367
00:15:19.362 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367
00:15:19.398 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:15:19.398 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)[L],70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179),b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)]}]
00:15:19.399 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=8, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=110), Traces: [0ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [2ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [2ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [9ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [10ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [12ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [12ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [24ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [29ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [30ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [32ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [32ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [64ms] refreshing cache from master, [65ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [70ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [72ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [74ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [74ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [104ms] refreshing cache from master, [105ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [109ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@1598053642(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@15f7c067 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1600906042 after retry RPC@1328601893) -> (continuation of Deferred@1069954512 after retry RPC@1986410750) -> (continuation of Deferred@1650546632 after retry RPC@263840982), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6d8fb89a -> wakeup thread Time-limited test -> (continuation of Deferred@1600906042 after retry RPC@1328601893) -> (continuation of Deferred@1069954512 after retry RPC@1986410750) -> (continuation of Deferred@1650546632 after retry RPC@263840982))) after lookup
00:15:19.400 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)] connecting to peer
00:15:19.401 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367
00:15:19.401 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)] cleaning up while in state CONNECTING due to: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367
00:15:19.401 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367
00:15:19.478 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:15:19.478 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)[L],70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179),b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)]}]
00:15:19.479 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=10, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=190), Traces: [0ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [2ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [2ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [9ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [10ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [12ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [12ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [24ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [29ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [30ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [32ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [32ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [64ms] refreshing cache from master, [65ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [70ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [72ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [74ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [74ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [104ms] refreshing cache from master, [105ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [109ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [111ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [113ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [113ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [184ms] refreshing cache from master, [184ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [189ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@1598053642(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@15f7c067 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1600906042 after retry RPC@1328601893) -> (continuation of Deferred@1069954512 after retry RPC@1986410750) -> (continuation of Deferred@1650546632 after retry RPC@263840982) -> (continuation of Deferred@319297933 after retry RPC@1398259822), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6d8fb89a -> wakeup thread Time-limited test -> (continuation of Deferred@1600906042 after retry RPC@1328601893) -> (continuation of Deferred@1069954512 after retry RPC@1986410750) -> (continuation of Deferred@1650546632 after retry RPC@263840982) -> (continuation of Deferred@319297933 after retry RPC@1398259822))) after lookup
00:15:19.480 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)] connecting to peer
00:15:19.481 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367
00:15:19.482 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)] cleaning up while in state CONNECTING due to: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367
00:15:19.482 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367
00:15:20.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:20.498482   667 raft_consensus.cc:493] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
00:15:20.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:20.499045   667 raft_consensus.cc:515] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:15:20.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:20.501991   667 leader_election.cc:290] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 094dbc44c6eb439083946b5ab0085ad0 (127.28.157.3:42367), 70a127fea97b464a89d03c5848cf6637 (127.28.157.1:37179)
00:15:20.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:20.506501   471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.157.3:42367: connect: Connection refused (error 111)
00:15:20.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:20.511620   471 leader_election.cc:336] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 094dbc44c6eb439083946b5ab0085ad0 (127.28.157.3:42367): Network error: Client connection negotiation failed: client connection to 127.28.157.3:42367: connect: Connection refused (error 111)
00:15:20.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:20.515637 32751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f4c1ce5aa4a4c8da912065e2decef6a" candidate_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" candidate_term: 9 candidate_status { last_received { term: 8 index: 32 } } ignore_live_leader: false dest_uuid: "70a127fea97b464a89d03c5848cf6637" is_pre_election: true
00:15:20.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:20.517524   469 leader_election.cc:304] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b5d8d89741dd4568a4c7b210a4da72d2; no voters: 094dbc44c6eb439083946b5ab0085ad0, 70a127fea97b464a89d03c5848cf6637
00:15:20.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:20.518364   667 raft_consensus.cc:2749] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
00:15:20.537 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:15:20.537 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)[L],70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179),b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)]}]
00:15:20.538 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=12, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=1249), Traces: [0ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [2ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [2ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [9ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [10ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [12ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [12ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [24ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [29ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [30ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [32ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [32ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [64ms] refreshing cache from master, [65ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [70ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [72ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [74ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [74ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [104ms] refreshing cache from master, [105ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [109ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [111ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [113ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [113ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [184ms] refreshing cache from master, [184ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [189ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [192ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [194ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [194ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [1244ms] refreshing cache from master, [1244ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [1248ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@1598053642(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@15f7c067 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1600906042 after retry RPC@1328601893) -> (continuation of Deferred@1069954512 after retry RPC@1986410750) -> (continuation of Deferred@1650546632 after retry RPC@263840982) -> (continuation of Deferred@319297933 after retry RPC@1398259822) -> (continuation of Deferred@1731573828 after retry RPC@176967804), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6d8fb89a -> wakeup thread Time-limited test -> (continuation of Deferred@1600906042 after retry RPC@1328601893) -> (continuation of Deferred@1069954512 after retry RPC@1986410750) -> (continuation of Deferred@1650546632 after retry RPC@263840982) -> (continuation of Deferred@319297933 after retry RPC@1398259822) -> (continuation of Deferred@1731573828 after retry RPC@176967804))) after lookup
00:15:20.538 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)] connecting to peer
00:15:20.540 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367
00:15:20.540 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)] cleaning up while in state CONNECTING due to: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367
00:15:20.540 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367
00:15:20.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:20.588716   674 raft_consensus.cc:493] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 094dbc44c6eb439083946b5ab0085ad0)
00:15:20.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:20.589284   674 raft_consensus.cc:515] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:15:20.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:20.591544   674 leader_election.cc:290] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b5d8d89741dd4568a4c7b210a4da72d2 (127.28.157.2:45951), 094dbc44c6eb439083946b5ab0085ad0 (127.28.157.3:42367)
00:15:20.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:20.599189 32622 leader_election.cc:336] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 094dbc44c6eb439083946b5ab0085ad0 (127.28.157.3:42367): Network error: Client connection negotiation failed: client connection to 127.28.157.3:42367: connect: Connection refused (error 111)
00:15:20.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:20.604143   579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f4c1ce5aa4a4c8da912065e2decef6a" candidate_uuid: "70a127fea97b464a89d03c5848cf6637" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" is_pre_election: true
00:15:20.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:20.605160   579 raft_consensus.cc:2468] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70a127fea97b464a89d03c5848cf6637 in term 8.
00:15:20.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:20.606966 32625 leader_election.cc:304] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 70a127fea97b464a89d03c5848cf6637, b5d8d89741dd4568a4c7b210a4da72d2; no voters: 094dbc44c6eb439083946b5ab0085ad0
00:15:20.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:20.607786   674 raft_consensus.cc:2804] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [term 9 FOLLOWER]: Leader pre-election won for term 10
00:15:20.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:20.608057   674 raft_consensus.cc:493] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 094dbc44c6eb439083946b5ab0085ad0)
00:15:20.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:20.608430   674 raft_consensus.cc:3060] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [term 9 FOLLOWER]: Advancing to term 10
00:15:20.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:20.614473   674 raft_consensus.cc:515] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:15:20.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:20.616140   674 leader_election.cc:290] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [CANDIDATE]: Term 10 election: Requested vote from peers b5d8d89741dd4568a4c7b210a4da72d2 (127.28.157.2:45951), 094dbc44c6eb439083946b5ab0085ad0 (127.28.157.3:42367)
00:15:20.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:20.617188   579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f4c1ce5aa4a4c8da912065e2decef6a" candidate_uuid: "70a127fea97b464a89d03c5848cf6637" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "b5d8d89741dd4568a4c7b210a4da72d2"
00:15:20.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:20.617856   579 raft_consensus.cc:3060] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [term 8 FOLLOWER]: Advancing to term 10
00:15:20.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:20.620008 32622 leader_election.cc:336] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 094dbc44c6eb439083946b5ab0085ad0 (127.28.157.3:42367): Network error: Client connection negotiation failed: client connection to 127.28.157.3:42367: connect: Connection refused (error 111)
00:15:20.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:20.624884   579 raft_consensus.cc:2468] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70a127fea97b464a89d03c5848cf6637 in term 10.
00:15:20.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:20.626035 32625 leader_election.cc:304] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 70a127fea97b464a89d03c5848cf6637, b5d8d89741dd4568a4c7b210a4da72d2; no voters: 094dbc44c6eb439083946b5ab0085ad0
00:15:20.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:20.626679   674 raft_consensus.cc:2804] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [term 10 FOLLOWER]: Leader election won for term 10
00:15:20.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:20.627827   674 raft_consensus.cc:697] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [term 10 LEADER]: Becoming Leader. State: Replica: 70a127fea97b464a89d03c5848cf6637, State: Running, Role: LEADER
00:15:20.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:20.628715   674 consensus_queue.cc:237] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 9.36, Last appended by leader: 36, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:15:20.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:20.639345 29607 catalog_manager.cc:5719] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 reported cstate change: term changed from 9 to 10, leader changed from 094dbc44c6eb439083946b5ab0085ad0 (127.28.157.3) to 70a127fea97b464a89d03c5848cf6637 (127.28.157.1). New cstate: current_term: 10 leader_uuid: "70a127fea97b464a89d03c5848cf6637" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } health_report { overall_health: HEALTHY } } }
00:15:21.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:21.079695   579 raft_consensus.cc:1275] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [term 10 FOLLOWER]: Refusing update from remote peer 70a127fea97b464a89d03c5848cf6637: Log matching property violated. Preceding OpId in replica: term: 8 index: 32. Preceding OpId from leader: term: 10 index: 37. (index mismatch)
00:15:21.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:21.081341   682 consensus_queue.cc:1048] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 32, Time since last communication: 0.001s
00:15:21.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:21.082109 32622 consensus_peers.cc:597] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 -> Peer 094dbc44c6eb439083946b5ab0085ad0 (127.28.157.3:42367): Couldn't send request to peer 094dbc44c6eb439083946b5ab0085ad0. Status: Network error: Client connection negotiation failed: client connection to 127.28.157.3:42367: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
00:15:21.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:21.632752 32622 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.157.3:42367: connect: Connection refused (error 111) [suppressed 3 similar messages]
00:15:22.356 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:15:22.357 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367),70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)[L],b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)]}]
00:15:22.357 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=14, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=3068), Traces: [0ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [2ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [2ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [9ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [10ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [12ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [12ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [24ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [29ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [30ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [32ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [32ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [64ms] refreshing cache from master, [65ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [70ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [72ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [74ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [74ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [104ms] refreshing cache from master, [105ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [109ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [111ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [113ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [113ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [184ms] refreshing cache from master, [184ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [189ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [192ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [194ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [194ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [1244ms] refreshing cache from master, [1244ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [1248ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [1250ms] sending RPC to server 094dbc44c6eb439083946b5ab0085ad0, [1252ms] received response from server 094dbc44c6eb439083946b5ab0085ad0: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [1252ms] delaying RPC due to: Network error: Failed to connect to peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367): Connection refused: /127.28.157.3:42367, [3064ms] refreshing cache from master, [3064ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [3068ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@1598053642(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@15f7c067 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1600906042 after retry RPC@1328601893) -> (continuation of Deferred@1069954512 after retry RPC@1986410750) -> (continuation of Deferred@1650546632 after retry RPC@263840982) -> (continuation of Deferred@319297933 after retry RPC@1398259822) -> (continuation of Deferred@1731573828 after retry RPC@176967804) -> (continuation of Deferred@483510346 after retry RPC@1949725724), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6d8fb89a -> wakeup thread Time-limited test -> (continuation of Deferred@1600906042 after retry RPC@1328601893) -> (continuation of Deferred@1069954512 after retry RPC@1986410750) -> (continuation of Deferred@1650546632 after retry RPC@263840982) -> (continuation of Deferred@319297933 after retry RPC@1398259822) -> (continuation of Deferred@1731573828 after retry RPC@176967804) -> (continuation of Deferred@483510346 after retry RPC@1949725724))) after lookup
00:15:22.358 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] connecting to peer
00:15:22.359 [DEBUG - kudu-nio-6] (Connection.java:757) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] Successfully connected to peer
00:15:22.386 [DEBUG - kudu-nio-6] (Negotiator.java:918) Authenticated connection [id: 0xa70d8ba6, L:/127.0.0.1:56846 - R:/127.28.157.1:37179] using TOKEN/null
00:15:22.446 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.28.157.3:42367
00:15:22.446 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

00:15:22.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:22.448406 29300 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu
00:15:22.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu
00:15:22.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-2/wal
00:15:22.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-2/data
00:15:22.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
00:15:22.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
00:15:22.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
00:15:22.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
00:15:22.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
00:15:22.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
00:15:22.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-2/logs
00:15:22.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-2/data/info.pb
00:15:22.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
00:15:22.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
00:15:22.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
00:15:22.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
00:15:22.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
00:15:22.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
00:15:22.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
00:15:22.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
00:15:22.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
00:15:22.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
00:15:22.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.28.157.3:42367
00:15:22.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.28.157.3
00:15:22.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.28.157.62:35965,127.28.157.61:44801,127.28.157.60:33393
00:15:22.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=36411
00:15:22.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.28.157.3
00:15:22.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.28.157.20:34671
00:15:22.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
00:15:22.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
00:15:22.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
00:15:22.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:22.800787   734 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
00:15:22.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:22.801373   734 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
00:15:22.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:22.802009   734 flags.cc:432] Enabled unsafe flag: --never_fsync=true
00:15:22.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:22.838563   734 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
00:15:22.839 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:22.839658   734 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.157.3
00:15:22.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:22.890522   734 tablet_server_runner.cc:78] Tablet server non-default flags:
00:15:22.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-2/logs
00:15:22.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
00:15:22.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
00:15:22.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
00:15:22.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.28.157.20:34671
00:15:22.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
00:15:22.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
00:15:22.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-2/data
00:15:22.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-2/wal
00:15:22.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
00:15:22.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
00:15:22.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.28.157.3:42367
00:15:22.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
00:15:22.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
00:15:22.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
00:15:22.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-2/data/info.pb
00:15:22.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.28.157.3
00:15:22.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=36411
00:15:22.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.28.157.62:35965,127.28.157.61:44801,127.28.157.60:33393
00:15:22.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
00:15:22.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
00:15:22.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
00:15:22.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
00:15:22.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
00:15:22.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.28.157.3
00:15:22.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
00:15:22.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
00:15:22.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
00:15:22.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision f7ed5b56c096f54e2f926bf94ceb7fc44c0c1329
00:15:22.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
00:15:22.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 11 Aug 2026 23:55:14 UTC on e7f111948823
00:15:22.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 17639
00:15:22.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
00:15:22.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:22.892388   734 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
00:15:22.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:22.894668   734 file_cache.cc:504] Constructed file cache file cache with capacity 419430
00:15:22.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:22.913086   746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
00:15:23.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:23.497831 32622 consensus_peers.cc:597] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 -> Peer 094dbc44c6eb439083946b5ab0085ad0 (127.28.157.3:42367): Couldn't send request to peer 094dbc44c6eb439083946b5ab0085ad0. Status: Network error: Client connection negotiation failed: client connection to 127.28.157.3:42367: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
00:15:24.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:22.915562   747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
00:15:24.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:24.315604   744 debug-util.cc:398] Leaking SignalData structure 0x7b080002ede0 after lost signal to thread 734
00:15:24.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:24.532794   744 kernel_stack_watchdog.cc:198] Thread 734 stuck at /home/jenkins-slave/workspace/build_and_test@4/src/kudu/util/thread.cc:642 for 399ms:
00:15:24.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Kernel stack:
00:15:24.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) (could not read kernel stack)
00:15:24.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) User stack:
00:15:24.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) <Timed out: thread did not respond: maybe it is blocking signals>
00:15:24.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:24.533617   734 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.620s	user 0.583s	sys 0.980s
00:15:24.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:24.534049   734 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.621s	user 0.583s	sys 0.980s
00:15:24.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:24.535166   750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
00:15:24.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:24.541303   748 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1626 milliseconds
00:15:24.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:24.541368   734 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
00:15:24.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:24.543439   734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
00:15:24.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:24.546721   734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
00:15:24.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:24.549510   734 hybrid_clock.cc:648] HybridClock initialized: now 1786493724549362 us; error 150 us; skew 500 ppm
00:15:24.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:24.550588   734 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
00:15:24.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:24.558027   734 webserver.cc:533] Webserver started at http://127.28.157.3:36411/ using document root <none> and password file <none>
00:15:24.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:24.559173   734 fs_manager.cc:362] Metadata directory not provided
00:15:24.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:24.559444   734 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
00:15:24.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:24.576825   734 fs_manager.cc:714] Time spent opening directory manager: real 0.010s	user 0.004s	sys 0.005s
00:15:24.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:24.583257   770 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
00:15:24.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:24.585316   734 fs_manager.cc:730] Time spent opening block manager: real 0.006s	user 0.005s	sys 0.000s
00:15:24.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:24.585933   734 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-2/data,/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-2/wal
00:15:24.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "094dbc44c6eb439083946b5ab0085ad0"
00:15:24.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 00:14:36 on dist-test-slave-t7g5"
00:15:24.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:24.588868   734 fs_report.cc:389] FS layout report
00:15:24.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
00:15:24.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-2/wal
00:15:24.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-2/wal
00:15:24.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-2/data/data
00:15:24.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
00:15:24.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
00:15:24.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
00:15:24.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
00:15:24.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
00:15:24.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
00:15:24.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
00:15:24.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
00:15:24.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
00:15:24.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
00:15:24.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
00:15:24.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:24.684090   734 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
00:15:24.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:24.686841   734 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
00:15:24.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:24.687525   734 kserver.cc:163] Server-wide thread pool size limit: 3276
00:15:24.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:24.691761   734 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
00:15:24.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:24.700641   784 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
00:15:24.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:24.710654   734 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
00:15:24.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:24.710934   734 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.013s	user 0.003s	sys 0.000s
00:15:24.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:24.711170   734 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
00:15:24.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:24.716405   734 ts_tablet_manager.cc:616] Registered 1 tablets
00:15:24.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:24.716634   734 ts_tablet_manager.cc:595] Time spent register tablets: real 0.005s	user 0.003s	sys 0.000s
00:15:24.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:24.717429   784 tablet_bootstrap.cc:492] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0: Bootstrap starting.
00:15:24.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:24.806185   784 log.cc:826] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0: Log is configured to *not* fsync() on all Append() calls
00:15:24.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:24.930503   734 rpc_server.cc:307] RPC server started. Bound to: 127.28.157.3:42367
00:15:24.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:24.930624   976 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.157.3:42367 every 8 connection(s)
00:15:24.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:24.934435   734 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-2/data/info.pb
00:15:24.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:24.939869 29300 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu as pid 734
00:15:24.942 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
00:15:24.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:24.951498   784 tablet_bootstrap.cc:492] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0: Bootstrap replayed 1/1 log segments. Stats: ops{read=36 overwritten=0 applied=36 ignored=0} inserts{seen=27 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
00:15:24.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:24.952723   784 tablet_bootstrap.cc:492] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0: Bootstrap complete.
00:15:24.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:24.954931   784 ts_tablet_manager.cc:1403] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0: Time spent bootstrapping tablet: real 0.238s	user 0.172s	sys 0.057s
00:15:24.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:24.979846   784 raft_consensus.cc:359] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 9 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:15:24.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:24.998746   784 raft_consensus.cc:740] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 9 FOLLOWER]: Becoming Follower/Learner. State: Replica: 094dbc44c6eb439083946b5ab0085ad0, State: Initialized, Role: FOLLOWER
00:15:25.002 [DEBUG - kudu-nio-6] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 30 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
00:15:25.003 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.28.157.1:37179
00:15:25.003 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

00:15:25.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:25.004158 29300 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu with pid 32581
00:15:25.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:25.010008   784 consensus_queue.cc:260] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 36, Last appended: 9.36, Last appended by leader: 36, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:15:25.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:25.036485   977 heartbeater.cc:344] Connected to a master server at 127.28.157.60:33393
00:15:25.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:25.037163   977 heartbeater.cc:461] Registering TS with master...
00:15:25.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:25.038985   977 heartbeater.cc:507] Master 127.28.157.60:33393 requested a full tablet report, sending...
00:15:25.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:25.041932   978 heartbeater.cc:344] Connected to a master server at 127.28.157.61:44801
00:15:25.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:25.042346   978 heartbeater.cc:461] Registering TS with master...
00:15:25.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:25.043388   978 heartbeater.cc:507] Master 127.28.157.61:44801 requested a full tablet report, sending...
00:15:25.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:25.036167   784 ts_tablet_manager.cc:1434] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0: Time spent starting tablet: real 0.073s	user 0.029s	sys 0.007s
00:15:25.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:25.047086 29608 ts_manager.cc:194] Re-registered known tserver with Master: 094dbc44c6eb439083946b5ab0085ad0 (127.28.157.3:42367)
00:15:25.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:25.048492 29485 ts_manager.cc:194] Re-registered known tserver with Master: 094dbc44c6eb439083946b5ab0085ad0 (127.28.157.3:42367)
00:15:25.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:25.050704 29608 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.28.157.3:51917
00:15:25.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:25.056417   977 heartbeater.cc:499] Master 127.28.157.60:33393 was elected leader, sending a full tablet report...
00:15:25.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:25.069044   984 heartbeater.cc:344] Connected to a master server at 127.28.157.62:35965
00:15:25.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:25.069469   984 heartbeater.cc:461] Registering TS with master...
00:15:25.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:25.070204   984 heartbeater.cc:507] Master 127.28.157.62:35965 requested a full tablet report, sending...
00:15:25.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:25.074116 29361 ts_manager.cc:194] Re-registered known tserver with Master: 094dbc44c6eb439083946b5ab0085ad0 (127.28.157.3:42367)
00:15:25.084 [DEBUG - kudu-nio-6] (Connection.java:239) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] handling channelInactive
00:15:25.084 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] cleaning up while in state READY due to: connection closed
00:15:25.084 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
00:15:25.085 [DEBUG - Time-limited test] (Connection.java:748) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] connecting to peer
00:15:25.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:25.085421   990 negotiation.cc:336] Failed RPC negotiation. Trace:
00:15:25.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:24.988152 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.28.157.1:50075 (local address 127.28.157.3:42367)
00:15:25.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:25.036561 (+ 48409us) server_negotiation.cc:207] Beginning negotiation
00:15:25.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:25.036601 (+    40us) server_negotiation.cc:400] Waiting for connection header
00:15:25.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:25.036682 (+    81us) server_negotiation.cc:408] Connection header received
00:15:25.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:25.036935 (+   253us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
00:15:25.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:25.036972 (+    37us) server_negotiation.cc:462] Received NEGOTIATE request from client
00:15:25.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:25.037452 (+   480us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
00:15:25.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:25.039466 (+  2014us) server_negotiation.cc:227] Negotiated authn=SASL
00:15:25.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:25.083364 (+ 43898us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.157.1:50075: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
00:15:25.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":40546,"thread_start_us":29852,"threads_started":1}
00:15:25.087 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:15:25.087 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] cleaning up while in state CONNECTING due to: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:15:25.087 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:15:25.098 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:15:25.098 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367),70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)[L],b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)]}]
00:15:25.098 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=2, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=13), Traces: [0ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [2ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [2ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [12ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@977328681(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@7d91dfe9 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@47fe3944 -> wakeup thread Time-limited test)) after lookup
00:15:25.099 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] connecting to peer
00:15:25.100 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:15:25.100 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] cleaning up while in state CONNECTING due to: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:15:25.101 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:15:25.117 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:15:25.118 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367),70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)[L],b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)]}]
00:15:25.118 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=4, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=33), Traces: [0ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [2ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [2ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [12ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [13ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [16ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [16ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [32ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@977328681(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@7d91dfe9 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@772728388 after retry RPC@403496664), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@47fe3944 -> wakeup thread Time-limited test -> (continuation of Deferred@772728388 after retry RPC@403496664))) after lookup
00:15:25.119 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] connecting to peer
00:15:25.120 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:15:25.121 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] cleaning up while in state CONNECTING due to: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:15:25.121 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:15:25.158 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:15:25.159 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367),70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)[L],b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)]}]
00:15:25.159 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=6, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=74), Traces: [0ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [2ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [2ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [12ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [13ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [16ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [16ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [32ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [33ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [36ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [36ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [67ms] refreshing cache from master, [68ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [73ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@977328681(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@7d91dfe9 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@772728388 after retry RPC@403496664) -> (continuation of Deferred@359403926 after retry RPC@1802041341), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@47fe3944 -> wakeup thread Time-limited test -> (continuation of Deferred@772728388 after retry RPC@403496664) -> (continuation of Deferred@359403926 after retry RPC@1802041341))) after lookup
00:15:25.160 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] connecting to peer
00:15:25.162 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:15:25.162 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] cleaning up while in state CONNECTING due to: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:15:25.162 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:15:25.237 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:15:25.237 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367),70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)[L],b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)]}]
00:15:25.238 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=8, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=153), Traces: [0ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [2ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [2ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [12ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [13ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [16ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [16ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [32ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [33ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [36ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [36ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [67ms] refreshing cache from master, [68ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [73ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [74ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [77ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [77ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [147ms] refreshing cache from master, [147ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [151ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@977328681(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@7d91dfe9 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@772728388 after retry RPC@403496664) -> (continuation of Deferred@359403926 after retry RPC@1802041341) -> (continuation of Deferred@1945217140 after retry RPC@299537697), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@47fe3944 -> wakeup thread Time-limited test -> (continuation of Deferred@772728388 after retry RPC@403496664) -> (continuation of Deferred@359403926 after retry RPC@1802041341) -> (continuation of Deferred@1945217140 after retry RPC@299537697))) after lookup
00:15:25.239 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] connecting to peer
00:15:25.241 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:15:25.241 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] cleaning up while in state CONNECTING due to: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:15:25.241 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:15:25.517 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:15:25.517 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367),70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)[L],b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)]}]
00:15:25.518 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=10, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=432), Traces: [0ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [2ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [2ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [12ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [13ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [16ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [16ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [32ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [33ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [36ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [36ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [67ms] refreshing cache from master, [68ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [73ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [74ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [77ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [77ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [147ms] refreshing cache from master, [147ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [151ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [153ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [156ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [156ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [427ms] refreshing cache from master, [427ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [431ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@977328681(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@7d91dfe9 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@772728388 after retry RPC@403496664) -> (continuation of Deferred@359403926 after retry RPC@1802041341) -> (continuation of Deferred@1945217140 after retry RPC@299537697) -> (continuation of Deferred@487606766 after retry RPC@1487320304), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@47fe3944 -> wakeup thread Time-limited test -> (continuation of Deferred@772728388 after retry RPC@403496664) -> (continuation of Deferred@359403926 after retry RPC@1802041341) -> (continuation of Deferred@1945217140 after retry RPC@299537697) -> (continuation of Deferred@487606766 after retry RPC@1487320304))) after lookup
00:15:25.518 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] connecting to peer
00:15:25.520 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:15:25.520 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] cleaning up while in state CONNECTING due to: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:15:25.520 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179) for tablet 2f4c1ce5aa4a4c8da912065e2decef6a: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179
00:15:26.468 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:26.467723  1003 raft_consensus.cc:493] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 70a127fea97b464a89d03c5848cf6637)
00:15:26.468 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:26.468182  1003 raft_consensus.cc:515] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:15:26.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:26.469889  1003 leader_election.cc:290] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 094dbc44c6eb439083946b5ab0085ad0 (127.28.157.3:42367), 70a127fea97b464a89d03c5848cf6637 (127.28.157.1:37179)
00:15:26.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:26.475488   469 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.157.1:37179: connect: Connection refused (error 111)
00:15:26.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:26.481642   469 leader_election.cc:336] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 70a127fea97b464a89d03c5848cf6637 (127.28.157.1:37179): Network error: Client connection negotiation failed: client connection to 127.28.157.1:37179: connect: Connection refused (error 111)
00:15:26.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:26.484702   907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f4c1ce5aa4a4c8da912065e2decef6a" candidate_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" candidate_term: 11 candidate_status { last_received { term: 10 index: 40 } } ignore_live_leader: false dest_uuid: "094dbc44c6eb439083946b5ab0085ad0" is_pre_election: true
00:15:26.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:26.485877   907 raft_consensus.cc:2468] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5d8d89741dd4568a4c7b210a4da72d2 in term 9.
00:15:26.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:26.487839   471 leader_election.cc:304] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 094dbc44c6eb439083946b5ab0085ad0, b5d8d89741dd4568a4c7b210a4da72d2; no voters: 70a127fea97b464a89d03c5848cf6637
00:15:26.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:26.488723  1003 raft_consensus.cc:2804] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [term 10 FOLLOWER]: Leader pre-election won for term 11
00:15:26.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:26.489049  1003 raft_consensus.cc:493] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 70a127fea97b464a89d03c5848cf6637)
00:15:26.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:26.489404  1003 raft_consensus.cc:3060] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [term 10 FOLLOWER]: Advancing to term 11
00:15:26.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:26.495381  1003 raft_consensus.cc:515] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:15:26.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:26.497146  1003 leader_election.cc:290] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [CANDIDATE]: Term 11 election: Requested vote from peers 094dbc44c6eb439083946b5ab0085ad0 (127.28.157.3:42367), 70a127fea97b464a89d03c5848cf6637 (127.28.157.1:37179)
00:15:26.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:26.498318   907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f4c1ce5aa4a4c8da912065e2decef6a" candidate_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" candidate_term: 11 candidate_status { last_received { term: 10 index: 40 } } ignore_live_leader: false dest_uuid: "094dbc44c6eb439083946b5ab0085ad0"
00:15:26.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:26.499078   907 raft_consensus.cc:3060] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 9 FOLLOWER]: Advancing to term 11
00:15:26.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:26.501750   469 leader_election.cc:336] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 70a127fea97b464a89d03c5848cf6637 (127.28.157.1:37179): Network error: Client connection negotiation failed: client connection to 127.28.157.1:37179: connect: Connection refused (error 111)
00:15:26.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:26.507058   907 raft_consensus.cc:2468] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5d8d89741dd4568a4c7b210a4da72d2 in term 11.
00:15:26.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:26.508366   471 leader_election.cc:304] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 094dbc44c6eb439083946b5ab0085ad0, b5d8d89741dd4568a4c7b210a4da72d2; no voters: 70a127fea97b464a89d03c5848cf6637
00:15:26.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:26.509428  1003 raft_consensus.cc:2804] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [term 11 FOLLOWER]: Leader election won for term 11
00:15:26.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:26.510995  1003 raft_consensus.cc:697] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [term 11 LEADER]: Becoming Leader. State: Replica: b5d8d89741dd4568a4c7b210a4da72d2, State: Running, Role: LEADER
00:15:26.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:26.512106  1003 consensus_queue.cc:237] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 10.40, Last appended by leader: 40, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:15:26.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:26.520884 29608 catalog_manager.cc:5719] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 reported cstate change: term changed from 10 to 11, leader changed from 70a127fea97b464a89d03c5848cf6637 (127.28.157.1) to b5d8d89741dd4568a4c7b210a4da72d2 (127.28.157.2). New cstate: current_term: 11 leader_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } health_report { overall_health: UNKNOWN } } }
00:15:26.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:26.951854   469 consensus_peers.cc:597] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 -> Peer 70a127fea97b464a89d03c5848cf6637 (127.28.157.1:37179): Couldn't send request to peer 70a127fea97b464a89d03c5848cf6637. Status: Network error: Client connection negotiation failed: client connection to 127.28.157.1:37179: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
00:15:27.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:27.022531   907 raft_consensus.cc:1275] T 2f4c1ce5aa4a4c8da912065e2decef6a P 094dbc44c6eb439083946b5ab0085ad0 [term 11 FOLLOWER]: Refusing update from remote peer b5d8d89741dd4568a4c7b210a4da72d2: Log matching property violated. Preceding OpId in replica: term: 9 index: 36. Preceding OpId from leader: term: 11 index: 41. (index mismatch)
00:15:27.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:27.024119  1003 consensus_queue.cc:1048] T 2f4c1ce5aa4a4c8da912065e2decef6a P b5d8d89741dd4568a4c7b210a4da72d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 36, Time since last communication: 0.000s
00:15:27.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:27.039420  1030 mvcc.cc:204] Tried to move back new op lower bound from 7317478286922280960 to 7317478279704285184. Current Snapshot: MvccSnapshot[applied={T|T < 7317478286922280960}]
00:15:27.096 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 2f4c1ce5aa4a4c8da912065e2decef6a for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353' with partition [<start>, <end>)
00:15:27.097 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2f4c1ce5aa4a4c8da912065e2decef6a@[094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367),70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179),b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)[L]]}]
00:15:27.097 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, attempt=12, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=2012), Traces: [0ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [2ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [2ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [12ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [13ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [16ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [16ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [32ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [33ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [36ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [36ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [67ms] refreshing cache from master, [68ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [73ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [74ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [77ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [77ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [147ms] refreshing cache from master, [147ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [151ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [153ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [156ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [156ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [427ms] refreshing cache from master, [427ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [431ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, [433ms] sending RPC to server 70a127fea97b464a89d03c5848cf6637, [435ms] received response from server 70a127fea97b464a89d03c5848cf6637: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [435ms] delaying RPC due to: Network error: Failed to connect to peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179): Connection refused: /127.28.157.1:37179, [2007ms] refreshing cache from master, [2007ms] Sub RPC GetTableLocations: sending RPC to server master-127.28.157.60:33393, [2011ms] Sub RPC GetTableLocations: received response from server master-127.28.157.60:33393: OK, deferred=Deferred@977328681(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$226/0x00007fba9c36ab50@7d91dfe9 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@772728388 after retry RPC@403496664) -> (continuation of Deferred@359403926 after retry RPC@1802041341) -> (continuation of Deferred@1945217140 after retry RPC@299537697) -> (continuation of Deferred@487606766 after retry RPC@1487320304) -> (continuation of Deferred@310584601 after retry RPC@2146374285), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@47fe3944 -> wakeup thread Time-limited test -> (continuation of Deferred@772728388 after retry RPC@403496664) -> (continuation of Deferred@359403926 after retry RPC@1802041341) -> (continuation of Deferred@1945217140 after retry RPC@299537697) -> (continuation of Deferred@487606766 after retry RPC@1487320304) -> (continuation of Deferred@310584601 after retry RPC@2146374285))) after lookup
00:15:27.098 [DEBUG - kudu-nio-2] (Connection.java:748) [peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)] connecting to peer
00:15:27.099 [DEBUG - kudu-nio-5] (Connection.java:757) [peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)] Successfully connected to peer
00:15:27.133 [DEBUG - kudu-nio-5] (Negotiator.java:918) Authenticated connection [id: 0x442ad9f8, L:/127.0.0.1:36442 - R:/127.28.157.2:45951] using TOKEN/null
00:15:27.185 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.28.157.1:37179
00:15:27.186 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

00:15:27.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:27.187536 29300 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu
00:15:27.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu
00:15:27.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/wal
00:15:27.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/data
00:15:27.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
00:15:27.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
00:15:27.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
00:15:27.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
00:15:27.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
00:15:27.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
00:15:27.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/logs
00:15:27.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/data/info.pb
00:15:27.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
00:15:27.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
00:15:27.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
00:15:27.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
00:15:27.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
00:15:27.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
00:15:27.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
00:15:27.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
00:15:27.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
00:15:27.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
00:15:27.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.28.157.1:37179
00:15:27.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.28.157.1
00:15:27.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.28.157.62:35965,127.28.157.61:44801,127.28.157.60:33393
00:15:27.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=35519
00:15:27.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.28.157.1
00:15:27.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.28.157.20:34671
00:15:27.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
00:15:27.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
00:15:27.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
00:15:27.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:27.582196  1045 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
00:15:27.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:27.582731  1045 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
00:15:27.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:27.583196  1045 flags.cc:432] Enabled unsafe flag: --never_fsync=true
00:15:27.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:27.621572  1045 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
00:15:27.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:27.622648  1045 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.157.1
00:15:27.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:27.663172  1045 tablet_server_runner.cc:78] Tablet server non-default flags:
00:15:27.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/logs
00:15:27.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
00:15:27.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
00:15:27.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
00:15:27.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.28.157.20:34671
00:15:27.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
00:15:27.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
00:15:27.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/data
00:15:27.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/wal
00:15:27.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
00:15:27.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
00:15:27.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.28.157.1:37179
00:15:27.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
00:15:27.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
00:15:27.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
00:15:27.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/data/info.pb
00:15:27.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.28.157.1
00:15:27.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=35519
00:15:27.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.28.157.62:35965,127.28.157.61:44801,127.28.157.60:33393
00:15:27.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
00:15:27.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
00:15:27.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
00:15:27.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
00:15:27.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
00:15:27.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.28.157.1
00:15:27.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
00:15:27.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
00:15:27.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
00:15:27.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision f7ed5b56c096f54e2f926bf94ceb7fc44c0c1329
00:15:27.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
00:15:27.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 11 Aug 2026 23:55:14 UTC on e7f111948823
00:15:27.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 17639
00:15:27.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
00:15:27.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:27.665045  1045 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
00:15:27.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:27.666980  1045 file_cache.cc:504] Constructed file cache file cache with capacity 419430
00:15:27.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:27.681248  1061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
00:15:28.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:28.423621   973 debug-util.cc:398] Leaking SignalData structure 0x7b08000b53e0 after lost signal to thread 736
00:15:28.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:28.425012   973 debug-util.cc:398] Leaking SignalData structure 0x7b08000b52e0 after lost signal to thread 976
00:15:29.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:27.682710  1062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
00:15:29.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:29.020309  1045 thread.cc:641] GCE (cloud detector) Time spent creating pthread: real 1.340s	user 0.588s	sys 0.744s
00:15:29.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:29.020689  1045 thread.cc:608] GCE (cloud detector) Time spent starting thread: real 1.341s	user 0.588s	sys 0.744s
00:15:29.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:29.023285  1045 server_base.cc:1061] running on GCE node
00:15:29.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:29.024266  1076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
00:15:29.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:29.025839  1045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
00:15:29.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:29.029799  1045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
00:15:29.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:29.031807  1045 hybrid_clock.cc:648] HybridClock initialized: now 1786493729031670 us; error 83 us; skew 500 ppm
00:15:29.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:29.033228  1045 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
00:15:29.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:29.043083  1045 webserver.cc:533] Webserver started at http://127.28.157.1:35519/ using document root <none> and password file <none>
00:15:29.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:29.044420  1045 fs_manager.cc:362] Metadata directory not provided
00:15:29.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:29.044708  1045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
00:15:29.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:29.057467  1045 fs_manager.cc:714] Time spent opening directory manager: real 0.008s	user 0.010s	sys 0.000s
00:15:29.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:29.065106  1084 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
00:15:29.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:29.067025  1045 fs_manager.cc:730] Time spent opening block manager: real 0.006s	user 0.005s	sys 0.002s
00:15:29.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:29.067483  1045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/data,/tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/wal
00:15:29.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "70a127fea97b464a89d03c5848cf6637"
00:15:29.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 00:14:32 on dist-test-slave-t7g5"
00:15:29.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:29.070463  1045 fs_report.cc:389] FS layout report
00:15:29.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
00:15:29.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/wal
00:15:29.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/wal
00:15:29.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/data/data
00:15:29.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
00:15:29.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
00:15:29.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
00:15:29.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
00:15:29.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
00:15:29.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
00:15:29.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
00:15:29.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
00:15:29.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
00:15:29.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
00:15:29.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
00:15:29.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:29.187994  1045 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
00:15:29.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:29.189574  1045 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
00:15:29.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:29.190045  1045 kserver.cc:163] Server-wide thread pool size limit: 3276
00:15:29.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:29.192925  1045 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
00:15:29.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:29.199724  1095 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
00:15:29.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:29.212390  1045 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
00:15:29.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:29.212692  1045 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.015s	user 0.000s	sys 0.002s
00:15:29.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:29.212994  1045 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
00:15:29.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:29.218196  1045 ts_tablet_manager.cc:616] Registered 1 tablets
00:15:29.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:29.218510  1045 ts_tablet_manager.cc:595] Time spent register tablets: real 0.006s	user 0.005s	sys 0.000s
00:15:29.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:29.218940  1095 tablet_bootstrap.cc:492] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637: Bootstrap starting.
00:15:29.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:29.310683  1095 log.cc:826] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637: Log is configured to *not* fsync() on all Append() calls
00:15:29.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:29.433336  1045 rpc_server.cc:307] RPC server started. Bound to: 127.28.157.1:37179
00:15:29.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:29.433785  1267 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.157.1:37179 every 8 connection(s)
00:15:29.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:29.437881  1045 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskq35Ux4/test-tmp/mini-kudu-cluster11128167211689354778/ts-0/data/info.pb
00:15:29.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:29.445376 29300 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu as pid 1045
00:15:29.446 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
00:15:29.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:29.464267  1095 tablet_bootstrap.cc:492] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637: Bootstrap replayed 1/1 log segments. Stats: ops{read=40 overwritten=0 applied=40 ignored=0} inserts{seen=30 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
00:15:29.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:29.465798  1095 tablet_bootstrap.cc:492] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637: Bootstrap complete.
00:15:29.468 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:29.467978  1095 ts_tablet_manager.cc:1403] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637: Time spent bootstrapping tablet: real 0.249s	user 0.183s	sys 0.046s
00:15:29.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:29.494686  1095 raft_consensus.cc:359] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [term 10 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:15:29.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:29.499266  1095 raft_consensus.cc:740] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [term 10 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70a127fea97b464a89d03c5848cf6637, State: Initialized, Role: FOLLOWER
00:15:29.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:29.501223  1095 consensus_queue.cc:260] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 40, Last appended: 10.40, Last appended by leader: 40, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5d8d89741dd4568a4c7b210a4da72d2" member_type: VOTER last_known_addr { host: "127.28.157.2" port: 45951 } } peers { permanent_uuid: "094dbc44c6eb439083946b5ab0085ad0" member_type: VOTER last_known_addr { host: "127.28.157.3" port: 42367 } } peers { permanent_uuid: "70a127fea97b464a89d03c5848cf6637" member_type: VOTER last_known_addr { host: "127.28.157.1" port: 37179 } }
00:15:29.561 [DEBUG - kudu-nio-5] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 33 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
00:15:29.576 [DEBUG - kudu-nio-5] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 33 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786493677353, tablet=2f4c1ce5aa4a4c8da912065e2decef6a, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
00:15:29.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:29.584283  1095 ts_tablet_manager.cc:1434] T 2f4c1ce5aa4a4c8da912065e2decef6a P 70a127fea97b464a89d03c5848cf6637: Time spent starting tablet: real 0.116s	user 0.066s	sys 0.041s
00:15:29.585 [DEBUG - main] (Connection.java:575) [peer master-127.28.157.60:33393(127.28.157.60:33393)] disconnecting while in state READY
00:15:29.586 [DEBUG - main] (Connection.java:575) [peer 70a127fea97b464a89d03c5848cf6637(127.28.157.1:37179)] disconnecting while in state TERMINATED
00:15:29.586 [DEBUG - main] (Connection.java:575) [peer master-127.28.157.61:44801(127.28.157.61:44801)] disconnecting while in state READY
00:15:29.586 [DEBUG - main] (Connection.java:575) [peer master-127.28.157.62:35965(127.28.157.62:35965)] disconnecting while in state READY
00:15:29.586 [DEBUG - main] (Connection.java:575) [peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)] disconnecting while in state READY
00:15:29.586 [DEBUG - main] (Connection.java:575) [peer 094dbc44c6eb439083946b5ab0085ad0(127.28.157.3:42367)] disconnecting while in state TERMINATED
00:15:29.588 [DEBUG - kudu-nio-2] (Connection.java:239) [peer master-127.28.157.60:33393(127.28.157.60:33393)] handling channelInactive
00:15:29.589 [DEBUG - kudu-nio-2] (Connection.java:709) [peer master-127.28.157.60:33393(127.28.157.60:33393)] cleaning up while in state READY due to: connection closed
00:15:29.590 [DEBUG - kudu-nio-5] (Connection.java:239) [peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)] handling channelInactive
00:15:29.590 [DEBUG - kudu-nio-5] (Connection.java:709) [peer b5d8d89741dd4568a4c7b210a4da72d2(127.28.157.2:45951)] cleaning up while in state READY due to: connection closed
00:15:29.590 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.28.157.61:44801(127.28.157.61:44801)] handling channelInactive
00:15:29.590 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.28.157.61:44801(127.28.157.61:44801)] cleaning up while in state READY due to: connection closed
00:15:29.591 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2704) Releasing all remaining resources
00:15:29.592 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.28.157.62:35965(127.28.157.62:35965)] handling channelInactive
00:15:29.592 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.28.157.62:35965(127.28.157.62:35965)] cleaning up while in state READY due to: connection closed
00:15:29.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:29.596822 29300 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu with pid 1045
00:15:29.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:29.651419  1287 negotiation.cc:336] Failed RPC negotiation. Trace:
00:15:29.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.475626 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.28.157.1:51575 (local address 127.28.157.62:35965)
00:15:29.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.520310 (+ 44684us) server_negotiation.cc:207] Beginning negotiation
00:15:29.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.520340 (+    30us) server_negotiation.cc:400] Waiting for connection header
00:15:29.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.520401 (+    61us) server_negotiation.cc:408] Connection header received
00:15:29.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.520569 (+   168us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
00:15:29.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.520598 (+    29us) server_negotiation.cc:462] Received NEGOTIATE request from client
00:15:29.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.520852 (+   254us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
00:15:29.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.521061 (+   209us) server_negotiation.cc:227] Negotiated authn=SASL
00:15:29.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.536868 (+ 15807us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
00:15:29.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.539607 (+  2739us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
00:15:29.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.561629 (+ 22022us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
00:15:29.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.562319 (+   690us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
00:15:29.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.649608 (+ 87289us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.157.1:51575: BlockingRecv error: recv got EOF from 127.28.157.1:51575 (error 108)
00:15:29.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":44013,"thread_start_us":46966,"threads_started":1}
00:15:29.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:29.651765  1289 negotiation.cc:336] Failed RPC negotiation. Trace:
00:15:29.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.498624 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.28.157.1:37179 (local address 127.28.157.2:60295)
00:15:29.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.536009 (+ 37385us) negotiation.cc:107] Waiting for socket to connect
00:15:29.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.536069 (+    60us) client_negotiation.cc:175] Beginning negotiation
00:15:29.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.536493 (+   424us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
00:15:29.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.550448 (+ 13955us) client_negotiation.cc:281] Received NEGOTIATE NegotiatePB response
00:15:29.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.550477 (+    29us) client_negotiation.cc:404] Received NEGOTIATE response from server
00:15:29.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.550641 (+   164us) client_negotiation.cc:190] Negotiated authn=SASL
00:15:29.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.551856 (+  1215us) client_negotiation.cc:530] Sending TLS_HANDSHAKE message to server
00:15:29.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.551945 (+    89us) client_negotiation.cc:262] Sending TLS_HANDSHAKE NegotiatePB request
00:15:29.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.570660 (+ 18715us) client_negotiation.cc:281] Received TLS_HANDSHAKE NegotiatePB response
00:15:29.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.570695 (+    35us) client_negotiation.cc:543] Received TLS_HANDSHAKE response from server
00:15:29.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.572046 (+  1351us) client_negotiation.cc:530] Sending TLS_HANDSHAKE message to server
00:15:29.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.572104 (+    58us) client_negotiation.cc:262] Sending TLS_HANDSHAKE NegotiatePB request
00:15:29.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.572519 (+   415us) client_negotiation.cc:564] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
00:15:29.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.610648 (+ 38129us) client_negotiation.cc:639] Initiating SASL PLAIN handshake
00:15:29.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.610688 (+    40us) client_negotiation.cc:672] calling sasl_client_start()
00:15:29.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.610767 (+    79us) client_negotiation.cc:830] callback for SASL_CB_AUTHNAME
00:15:29.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.610941 (+   174us) client_negotiation.cc:262] Sending SASL_INITIATE NegotiatePB request
00:15:29.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.649513 (+ 38572us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.157.1:37179: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
00:15:29.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"client-negotiator.queue_time_us":36648,"thread_start_us":36314,"threads_started":1}
00:15:29.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:29.654476 29300 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu with pid 434
00:15:29.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:29.657546  1278 negotiation.cc:336] Failed RPC negotiation. Trace:
00:15:29.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.455372 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.28.157.1:35153 (local address 127.28.157.60:33393)
00:15:29.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.544269 (+ 88897us) server_negotiation.cc:207] Beginning negotiation
00:15:29.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.544324 (+    55us) server_negotiation.cc:400] Waiting for connection header
00:15:29.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.544410 (+    86us) server_negotiation.cc:408] Connection header received
00:15:29.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.544664 (+   254us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
00:15:29.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.544693 (+    29us) server_negotiation.cc:462] Received NEGOTIATE request from client
00:15:29.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.545000 (+   307us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
00:15:29.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.575507 (+ 30507us) server_negotiation.cc:227] Negotiated authn=SASL
00:15:29.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.576032 (+   525us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
00:15:29.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.578125 (+  2093us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
00:15:29.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.585560 (+  7435us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
00:15:29.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.586202 (+   642us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
00:15:29.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.614602 (+ 28400us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
00:15:29.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.614851 (+   249us) server_negotiation.cc:882] Received SASL_INITIATE request from client
00:15:29.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.614891 (+    40us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
00:15:29.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.614978 (+    87us) server_negotiation.cc:923] calling sasl_server_start()
00:15:29.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.615100 (+   122us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
00:15:29.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.615174 (+    74us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
00:15:29.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.615339 (+   165us) server_negotiation.cc:1036] Waiting for connection context
00:15:29.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.649537 (+ 34198us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.157.1:35153: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
00:15:29.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":87875,"thread_start_us":87404,"threads_started":1}
00:15:29.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 00:15:29.658826  1280 negotiation.cc:336] Failed RPC negotiation. Trace:
00:15:29.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.456847 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.28.157.1:47665 (local address 127.28.157.61:44801)
00:15:29.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.547854 (+ 91007us) server_negotiation.cc:207] Beginning negotiation
00:15:29.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.547891 (+    37us) server_negotiation.cc:400] Waiting for connection header
00:15:29.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.547967 (+    76us) server_negotiation.cc:408] Connection header received
00:15:29.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.548202 (+   235us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
00:15:29.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.548232 (+    30us) server_negotiation.cc:462] Received NEGOTIATE request from client
00:15:29.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.548518 (+   286us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
00:15:29.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.567708 (+ 19190us) server_negotiation.cc:227] Negotiated authn=SASL
00:15:29.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.568233 (+   525us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
00:15:29.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.570211 (+  1978us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
00:15:29.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.613435 (+ 43224us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
00:15:29.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.614153 (+   718us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
00:15:29.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 00:15:29.650378 (+ 36225us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.157.1:47665: BlockingRecv error: recv got EOF from 127.28.157.1:47665 (error 108)
00:15:29.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":90236,"thread_start_us":89869,"threads_started":1}
00:15:29.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:29.702543 29300 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu with pid 734
00:15:29.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:29.744572 29300 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu with pid 29314
00:15:29.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:29.799299 29300 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu with pid 29419
00:15:29.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 00:15:29.846444 29300 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskq35Ux4/build/tsan/bin/kudu with pid 29542
00:15:31.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-08-12T00:15:31Z chronyd exiting

Time: 70.481

OK (1 test)

