JUnit version 4.13.2
.18:21:37.593 [DEBUG - main] (RandomUtils.java:49) Using random seed: 1786472497586
18:21:37.641 [INFO - main] (RetryRule.java:138) Creating RetryStatement with result reporter and retry count of 0 (no retries)
18:21:37.669 [INFO - main] (KuduTestHarness.java:155) Creating a new MiniKuduCluster...
18:21:37.679 [INFO - main] (TempDirUtils.java:71) Using the temp directory defined by TEST_TMPDIR: /tmp/dist-test-taskpuxqmP/test-tmp
18:21:37.702 [INFO - main] (KuduBinaryLocator.java:83) Using Kudu binary directory specified by system property 'kuduBinDir': ../../../build/tsan/bin
18:21:37.705 [INFO - main] (MiniKuduCluster.java:217) Starting process: [/tmp/dist-test-taskpuxqmP/java/build/dist-test/../../../build/tsan/bin/kudu, test, mini_cluster, --serialization=pb]
18:21:37.906 [DEBUG - main] (MiniKuduCluster.java:185) Request: create_cluster {
  num_masters: 3
  num_tservers: 3
  enable_kerberos: false
  cluster_root: "/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744"
  hms_mode: NONE
  mini_kdc_options {
  }
  principal: "kudu"
  mini_oidc_options {
  }
}

18:21:38.076 [DEBUG - main] (MiniKuduCluster.java:195) Response: 
18:21:38.078 [DEBUG - main] (MiniKuduCluster.java:185) Request: start_cluster {
}

18:21:38.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:38.081662 19386 mini_oidc.cc:150] Starting JWKS server
18:21:38.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:38.085053 19386 webserver.cc:533] Webserver started at http://127.0.0.1:33783/ using document root <none> and password file <none>
18:21:38.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:38.085542 19386 mini_oidc.cc:186] Starting OIDC Discovery server
18:21:38.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:38.087906 19386 webserver.cc:533] Webserver started at http://127.0.0.1:36459/ using document root <none> and password file <none>
18:21:38.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-08-11T18:21:38Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
18:21:38.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-08-11T18:21:38Z Disabled control of system clock
18:21:38.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:38.146695 19386 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu
18:21:38.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu
18:21:38.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/master-0/wal
18:21:38.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/master-0/data
18:21:38.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
18:21:38.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
18:21:38.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
18:21:38.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:21:38.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:21:38.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:21:38.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/master-0/logs
18:21:38.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/master-0/data/info.pb
18:21:38.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:21:38.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
18:21:38.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
18:21:38.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
18:21:38.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
18:21:38.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:21:38.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:21:38.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:21:38.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
18:21:38.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
18:21:38.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
18:21:38.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
18:21:38.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.18.238.190:40173
18:21:38.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.18.238.190
18:21:38.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
18:21:38.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.18.238.148:35877
18:21:38.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:21:38.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:21:38.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
18:21:38.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
18:21:38.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.18.238.190:40173,127.18.238.189:35025,127.18.238.188:40453
18:21:38.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
18:21:38.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:33783/jwks.json with env {}
18:21:38.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:38.433101 19402 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
18:21:38.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:38.433591 19402 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
18:21:38.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:38.434072 19402 flags.cc:432] Enabled unsafe flag: --never_fsync=true
18:21:38.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:38.463361 19402 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
18:21:38.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:38.463617 19402 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
18:21:38.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:38.463806 19402 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
18:21:38.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:38.464002 19402 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
18:21:38.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:38.464182 19402 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
18:21:38.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:38.464412 19402 flags.cc:432] Enabled experimental flag: --jwks_url=http://localhost:33783/jwks.json
18:21:38.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:38.498224 19402 master_runner.cc:387] Master server non-default flags:
18:21:38.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/master-0/logs
18:21:38.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:21:38.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
18:21:38.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:21:38.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.18.238.148:35877
18:21:38.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:21:38.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
18:21:38.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/master-0/data
18:21:38.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/master-0/wal
18:21:38.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
18:21:38.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.18.238.190:40173,127.18.238.189:35025,127.18.238.188:40453
18:21:38.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:21:38.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:21:38.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
18:21:38.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.18.238.190:40173
18:21:38.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
18:21:38.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
18:21:38.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
18:21:38.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:33783/jwks.json
18:21:38.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:21:38.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:21:38.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/master-0/data/info.pb
18:21:38.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.18.238.190
18:21:38.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
18:21:38.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
18:21:38.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:21:38.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
18:21:38.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
18:21:38.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:21:38.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
18:21:38.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
18:21:38.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
18:21:38.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision cfdbd367ce7fd9747cb6f1860fc64263d9f9f79a
18:21:38.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
18:21:38.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 11 Aug 2026 18:01:34 UTC on bdcb31816ec0
18:21:38.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 17632
18:21:38.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
18:21:38.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:38.499583 19402 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
18:21:38.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:38.501128 19402 file_cache.cc:504] Constructed file cache file cache with capacity 419430
18:21:38.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:38.512203 19409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:21:39.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:38.512216 19410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:21:39.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:39.684733 19411 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1170 milliseconds
18:21:39.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:39.687106 19412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:21:39.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:39.690596 19402 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.179s	user 0.334s	sys 0.844s
18:21:39.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:39.690887 19402 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.179s	user 0.334s	sys 0.844s
18:21:39.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:39.691290 19402 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
18:21:39.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:39.693028 19402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
18:21:39.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:39.695931 19402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
18:21:39.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:39.697325 19402 hybrid_clock.cc:648] HybridClock initialized: now 1786472499697314 us; error 93 us; skew 500 ppm
18:21:39.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:39.698377 19402 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
18:21:39.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:39.705360 19402 webserver.cc:533] Webserver started at http://127.18.238.190:39133/ using document root <none> and password file <none>
18:21:39.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:39.706465 19402 fs_manager.cc:362] Metadata directory not provided
18:21:39.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:39.706717 19402 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
18:21:39.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:39.707306 19402 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
18:21:39.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:39.711534 19402 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/master-0/data/instance:
18:21:39.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "3ede11d6ee67428fb7223a5c9bb41382"
18:21:39.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 18:21:39 on dist-test-slave-6nmv"
18:21:39.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:39.712955 19402 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/master-0/wal/instance:
18:21:39.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "3ede11d6ee67428fb7223a5c9bb41382"
18:21:39.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 18:21:39 on dist-test-slave-6nmv"
18:21:39.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:39.721310 19402 fs_manager.cc:696] Time spent creating directory manager: real 0.008s	user 0.010s	sys 0.000s
18:21:39.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:39.727180 19419 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
18:21:39.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:39.728595 19402 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.001s
18:21:39.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:39.728998 19402 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/master-0/data,/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/master-0/wal
18:21:39.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "3ede11d6ee67428fb7223a5c9bb41382"
18:21:39.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 18:21:39 on dist-test-slave-6nmv"
18:21:39.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:39.729406 19402 fs_report.cc:389] FS layout report
18:21:39.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
18:21:39.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/master-0/wal
18:21:39.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/master-0/wal
18:21:39.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/master-0/data/data
18:21:39.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
18:21:39.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
18:21:39.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
18:21:39.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
18:21:39.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
18:21:39.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
18:21:39.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
18:21:39.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
18:21:39.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
18:21:39.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
18:21:39.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
18:21:39.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:39.798653 19402 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
18:21:39.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:39.800231 19402 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
18:21:39.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:39.800731 19402 kserver.cc:163] Server-wide thread pool size limit: 3276
18:21:39.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:39.869385 19402 rpc_server.cc:307] RPC server started. Bound to: 127.18.238.190:40173
18:21:39.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:39.869468 19478 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.238.190:40173 every 8 connection(s)
18:21:39.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:39.872387 19402 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/master-0/data/info.pb
18:21:39.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:39.878405 19479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
18:21:39.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:39.882475 19386 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu as pid 19402
18:21:39.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:39.883008 19386 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/master-0/wal/instance
18:21:39.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:39.888051 19386 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu
18:21:39.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu
18:21:39.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/master-1/wal
18:21:39.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/master-1/data
18:21:39.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
18:21:39.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
18:21:39.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
18:21:39.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:21:39.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:21:39.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:21:39.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/master-1/logs
18:21:39.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/master-1/data/info.pb
18:21:39.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:21:39.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
18:21:39.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
18:21:39.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
18:21:39.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
18:21:39.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:21:39.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:21:39.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:21:39.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
18:21:39.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
18:21:39.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
18:21:39.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
18:21:39.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.18.238.189:35025
18:21:39.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.18.238.189
18:21:39.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
18:21:39.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.18.238.148:35877
18:21:39.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:21:39.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:21:39.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
18:21:39.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
18:21:39.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.18.238.190:40173,127.18.238.189:35025,127.18.238.188:40453
18:21:39.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
18:21:39.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:33783/jwks.json with env {}
18:21:39.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:39.893178 19479 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.238.190" port: 40173 } has no permanent_uuid. Determining permanent_uuid...
18:21:39.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:39.918125 19479 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.238.189" port: 35025 } has no permanent_uuid. Determining permanent_uuid...
18:21:39.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:39.920624 19421 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.238.189:35025: connect: Connection refused (error 111)
18:21:39.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:39.924247 19479 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.238.189:35025: Network error: Client connection negotiation failed: client connection to 127.18.238.189:35025: connect: Connection refused (error 111)
18:21:39.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:39.973686 19479 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.238.189" port: 35025 } attempt: 1
18:21:39.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:39.978763 19479 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.238.189:35025: Network error: Client connection negotiation failed: client connection to 127.18.238.189:35025: connect: Connection refused (error 111)
18:21:40.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:40.047215 19479 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.238.189" port: 35025 } attempt: 2
18:21:40.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:40.052126 19479 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.238.189:35025: Network error: Client connection negotiation failed: client connection to 127.18.238.189:35025: connect: Connection refused (error 111)
18:21:40.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:40.143529 19479 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.238.189" port: 35025 } attempt: 3
18:21:40.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:40.147748 19479 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.238.189:35025: Network error: Client connection negotiation failed: client connection to 127.18.238.189:35025: connect: Connection refused (error 111)
18:21:40.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:40.198050 19482 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
18:21:40.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:40.198647 19482 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
18:21:40.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:40.199352 19482 flags.cc:432] Enabled unsafe flag: --never_fsync=true
18:21:40.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:40.239039 19482 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
18:21:40.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:40.239310 19482 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
18:21:40.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:40.239508 19482 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
18:21:40.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:40.239715 19482 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
18:21:40.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:40.239882 19482 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
18:21:40.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:40.240006 19482 flags.cc:432] Enabled experimental flag: --jwks_url=http://localhost:33783/jwks.json
18:21:40.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:40.272653 19482 master_runner.cc:387] Master server non-default flags:
18:21:40.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/master-1/logs
18:21:40.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:21:40.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
18:21:40.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:21:40.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.18.238.148:35877
18:21:40.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:21:40.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
18:21:40.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/master-1/data
18:21:40.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/master-1/wal
18:21:40.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
18:21:40.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.18.238.190:40173,127.18.238.189:35025,127.18.238.188:40453
18:21:40.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:21:40.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:21:40.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
18:21:40.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.18.238.189:35025
18:21:40.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
18:21:40.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
18:21:40.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
18:21:40.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:33783/jwks.json
18:21:40.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:21:40.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:21:40.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/master-1/data/info.pb
18:21:40.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.18.238.189
18:21:40.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
18:21:40.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
18:21:40.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:21:40.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
18:21:40.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
18:21:40.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:21:40.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
18:21:40.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
18:21:40.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
18:21:40.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision cfdbd367ce7fd9747cb6f1860fc64263d9f9f79a
18:21:40.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
18:21:40.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 11 Aug 2026 18:01:34 UTC on bdcb31816ec0
18:21:40.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 17632
18:21:40.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
18:21:40.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:40.274055 19482 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
18:21:40.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:40.275748 19482 file_cache.cc:504] Constructed file cache file cache with capacity 419430
18:21:40.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:40.288046 19493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:21:40.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:40.291132 19479 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.238.189" port: 35025 } attempt: 4
18:21:40.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:40.297765 19479 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.238.189:35025: Network error: Client connection negotiation failed: client connection to 127.18.238.189:35025: connect: Connection refused (error 111)
18:21:40.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:40.597226 19479 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.238.189" port: 35025 } attempt: 5
18:21:40.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:40.619042 19479 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.238.189:35025: Network error: Client connection negotiation failed: client connection to 127.18.238.189:35025: connect: Connection refused (error 111)
18:21:41.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:41.166625 19479 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.238.189" port: 35025 } attempt: 6
18:21:41.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:41.192519 19479 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.238.189:35025: Network error: Client connection negotiation failed: client connection to 127.18.238.189:35025: connect: Connection refused (error 111)
18:21:41.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:41.693228 19492 debug-util.cc:398] Leaking SignalData structure 0x7b080002ee00 after lost signal to thread 19482
18:21:41.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:41.708689 19482 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.420s	user 0.531s	sys 0.888s
18:21:41.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:41.709118 19482 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.421s	user 0.531s	sys 0.888s
18:21:41.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:40.288766 19494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:21:41.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:41.710884 19496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:21:41.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:41.714838 19495 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1424 milliseconds
18:21:41.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:41.714906 19482 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
18:21:41.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:41.716217 19482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
18:21:41.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:41.718760 19482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
18:21:41.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:41.720129 19482 hybrid_clock.cc:648] HybridClock initialized: now 1786472501720088 us; error 35 us; skew 500 ppm
18:21:41.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:41.720968 19482 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
18:21:41.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:41.726653 19482 webserver.cc:533] Webserver started at http://127.18.238.189:45359/ using document root <none> and password file <none>
18:21:41.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:41.727679 19482 fs_manager.cc:362] Metadata directory not provided
18:21:41.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:41.727943 19482 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
18:21:41.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:41.728482 19482 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
18:21:41.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:41.732646 19482 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/master-1/data/instance:
18:21:41.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "9ffbce142df84608a495ea1fb36df8d1"
18:21:41.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 18:21:41 on dist-test-slave-6nmv"
18:21:41.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:41.733970 19482 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/master-1/wal/instance:
18:21:41.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "9ffbce142df84608a495ea1fb36df8d1"
18:21:41.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 18:21:41 on dist-test-slave-6nmv"
18:21:41.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:41.741171 19482 fs_manager.cc:696] Time spent creating directory manager: real 0.006s	user 0.005s	sys 0.001s
18:21:41.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:41.746537 19505 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
18:21:41.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:41.747571 19482 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.002s
18:21:41.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:41.747951 19482 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/master-1/data,/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/master-1/wal
18:21:41.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "9ffbce142df84608a495ea1fb36df8d1"
18:21:41.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 18:21:41 on dist-test-slave-6nmv"
18:21:41.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:41.748304 19482 fs_report.cc:389] FS layout report
18:21:41.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
18:21:41.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/master-1/wal
18:21:41.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/master-1/wal
18:21:41.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/master-1/data/data
18:21:41.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
18:21:41.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
18:21:41.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
18:21:41.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
18:21:41.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
18:21:41.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
18:21:41.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
18:21:41.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
18:21:41.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
18:21:41.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
18:21:41.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
18:21:41.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:41.797569 19482 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
18:21:41.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:41.799167 19482 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
18:21:41.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:41.799695 19482 kserver.cc:163] Server-wide thread pool size limit: 3276
18:21:41.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:41.865453 19482 rpc_server.cc:307] RPC server started. Bound to: 127.18.238.189:35025
18:21:41.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:41.865527 19566 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.238.189:35025 every 8 connection(s)
18:21:41.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:41.868463 19482 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/master-1/data/info.pb
18:21:41.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:41.874696 19386 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu as pid 19482
18:21:41.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:41.875167 19386 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/master-1/wal/instance
18:21:41.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:41.875160 19567 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
18:21:41.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:41.878083 19386 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu
18:21:41.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu
18:21:41.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/master-2/wal
18:21:41.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/master-2/data
18:21:41.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
18:21:41.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
18:21:41.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
18:21:41.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:21:41.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:21:41.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:21:41.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/master-2/logs
18:21:41.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/master-2/data/info.pb
18:21:41.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:21:41.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
18:21:41.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
18:21:41.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
18:21:41.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
18:21:41.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:21:41.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:21:41.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:21:41.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
18:21:41.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
18:21:41.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
18:21:41.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
18:21:41.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.18.238.188:40453
18:21:41.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.18.238.188
18:21:41.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
18:21:41.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.18.238.148:35877
18:21:41.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:21:41.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:21:41.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
18:21:41.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
18:21:41.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.18.238.190:40173,127.18.238.189:35025,127.18.238.188:40453
18:21:41.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
18:21:41.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:33783/jwks.json with env {}
18:21:41.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:41.888661 19567 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.238.190" port: 40173 } has no permanent_uuid. Determining permanent_uuid...
18:21:41.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:41.908341 19567 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.238.189" port: 35025 } has no permanent_uuid. Determining permanent_uuid...
18:21:41.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:41.922408 19567 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.238.188" port: 40453 } has no permanent_uuid. Determining permanent_uuid...
18:21:41.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:41.924467 19506 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.238.188:40453: connect: Connection refused (error 111)
18:21:41.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:41.928527 19567 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.238.188:40453: Network error: Client connection negotiation failed: client connection to 127.18.238.188:40453: connect: Connection refused (error 111)
18:21:41.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:41.977958 19567 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.238.188" port: 40453 } attempt: 1
18:21:41.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:41.982774 19567 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.238.188:40453: Network error: Client connection negotiation failed: client connection to 127.18.238.188:40453: connect: Connection refused (error 111)
18:21:42.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:42.051208 19567 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.238.188" port: 40453 } attempt: 2
18:21:42.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:42.055434 19567 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.238.188:40453: Network error: Client connection negotiation failed: client connection to 127.18.238.188:40453: connect: Connection refused (error 111)
18:21:42.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:42.146874 19567 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.238.188" port: 40453 } attempt: 3
18:21:42.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:42.150995 19567 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.238.188:40453: Network error: Client connection negotiation failed: client connection to 127.18.238.188:40453: connect: Connection refused (error 111)
18:21:42.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:42.172501 19569 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
18:21:42.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:42.173098 19569 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
18:21:42.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:42.173593 19569 flags.cc:432] Enabled unsafe flag: --never_fsync=true
18:21:42.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:42.202653 19569 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
18:21:42.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:42.202889 19569 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
18:21:42.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:42.203063 19569 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
18:21:42.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:42.203223 19569 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
18:21:42.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:42.203436 19569 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
18:21:42.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:42.203641 19569 flags.cc:432] Enabled experimental flag: --jwks_url=http://localhost:33783/jwks.json
18:21:42.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:42.236397 19569 master_runner.cc:387] Master server non-default flags:
18:21:42.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/master-2/logs
18:21:42.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:21:42.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
18:21:42.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:21:42.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.18.238.148:35877
18:21:42.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:21:42.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
18:21:42.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/master-2/data
18:21:42.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/master-2/wal
18:21:42.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
18:21:42.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.18.238.190:40173,127.18.238.189:35025,127.18.238.188:40453
18:21:42.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:21:42.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:21:42.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
18:21:42.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.18.238.188:40453
18:21:42.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
18:21:42.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
18:21:42.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
18:21:42.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:33783/jwks.json
18:21:42.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:21:42.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:21:42.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/master-2/data/info.pb
18:21:42.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.18.238.188
18:21:42.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
18:21:42.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
18:21:42.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:21:42.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
18:21:42.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
18:21:42.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:21:42.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
18:21:42.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
18:21:42.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
18:21:42.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision cfdbd367ce7fd9747cb6f1860fc64263d9f9f79a
18:21:42.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
18:21:42.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 11 Aug 2026 18:01:34 UTC on bdcb31816ec0
18:21:42.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 17632
18:21:42.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
18:21:42.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:42.237695 19569 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
18:21:42.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:42.239708 19569 file_cache.cc:504] Constructed file cache file cache with capacity 419430
18:21:42.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:42.251430 19581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:21:42.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:42.253079 19479 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.238.189" port: 35025 } attempt: 7
18:21:42.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:42.281277 19479 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.238.188" port: 40453 } has no permanent_uuid. Determining permanent_uuid...
18:21:42.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:42.294395 19567 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.238.188" port: 40453 } attempt: 4
18:21:42.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:42.301764 19567 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.238.188:40453: Network error: Client connection negotiation failed: client connection to 127.18.238.188:40453: connect: Connection refused (error 111)
18:21:42.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:42.315840 19479 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.238.188:40453: Network error: Client connection negotiation failed: client connection to 127.18.238.188:40453: connect: Connection refused (error 111)
18:21:42.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:42.374372 19479 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.238.188" port: 40453 } attempt: 1
18:21:42.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:42.382961 19479 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.238.188:40453: Network error: Client connection negotiation failed: client connection to 127.18.238.188:40453: connect: Connection refused (error 111)
18:21:42.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:42.464493 19479 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.238.188" port: 40453 } attempt: 2
18:21:42.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:42.473383 19479 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.238.188:40453: Network error: Client connection negotiation failed: client connection to 127.18.238.188:40453: connect: Connection refused (error 111)
18:21:42.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:42.558909 19479 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.238.188" port: 40453 } attempt: 3
18:21:42.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:42.567785 19479 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.238.188:40453: Network error: Client connection negotiation failed: client connection to 127.18.238.188:40453: connect: Connection refused (error 111)
18:21:42.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:42.601285 19567 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.238.188" port: 40453 } attempt: 5
18:21:42.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:42.620457 19567 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.238.188:40453: Network error: Client connection negotiation failed: client connection to 127.18.238.188:40453: connect: Connection refused (error 111)
18:21:42.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:42.708341 19479 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.238.188" port: 40453 } attempt: 4
18:21:42.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:42.718858 19479 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.238.188:40453: Network error: Client connection negotiation failed: client connection to 127.18.238.188:40453: connect: Connection refused (error 111)
18:21:43.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:43.002405 19479 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.238.188" port: 40453 } attempt: 5
18:21:43.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:43.011612 19479 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.238.188:40453: Network error: Client connection negotiation failed: client connection to 127.18.238.188:40453: connect: Connection refused (error 111)
18:21:43.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:43.168081 19567 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.238.188" port: 40453 } attempt: 6
18:21:43.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:43.193725 19567 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.238.188:40453: Network error: Client connection negotiation failed: client connection to 127.18.238.188:40453: connect: Connection refused (error 111)
18:21:43.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:42.251713 19582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:21:43.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:43.456184 19584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:21:43.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:43.458243 19583 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1205 milliseconds
18:21:43.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:43.458797 19569 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.207s	user 0.326s	sys 0.841s
18:21:43.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:43.459123 19569 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.208s	user 0.326s	sys 0.841s
18:21:43.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:43.459403 19569 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
18:21:43.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:43.460933 19569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
18:21:43.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:43.463925 19569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
18:21:43.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:43.465657 19569 hybrid_clock.cc:648] HybridClock initialized: now 1786472503465542 us; error 90 us; skew 500 ppm
18:21:43.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:43.466751 19569 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
18:21:43.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:43.475063 19569 webserver.cc:533] Webserver started at http://127.18.238.188:40865/ using document root <none> and password file <none>
18:21:43.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:43.476368 19569 fs_manager.cc:362] Metadata directory not provided
18:21:43.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:43.476646 19569 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
18:21:43.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:43.477272 19569 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
18:21:43.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:43.483922 19569 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/master-2/data/instance:
18:21:43.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "86bb2b3f4c994cbf9341c7bcb469f41d"
18:21:43.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 18:21:43 on dist-test-slave-6nmv"
18:21:43.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:43.485440 19569 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/master-2/wal/instance:
18:21:43.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "86bb2b3f4c994cbf9341c7bcb469f41d"
18:21:43.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 18:21:43 on dist-test-slave-6nmv"
18:21:43.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:43.495163 19569 fs_manager.cc:696] Time spent creating directory manager: real 0.009s	user 0.006s	sys 0.002s
18:21:43.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:43.502803 19600 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
18:21:43.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:43.504010 19569 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.001s	sys 0.004s
18:21:43.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:43.504374 19569 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/master-2/data,/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/master-2/wal
18:21:43.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "86bb2b3f4c994cbf9341c7bcb469f41d"
18:21:43.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 18:21:43 on dist-test-slave-6nmv"
18:21:43.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:43.504776 19569 fs_report.cc:389] FS layout report
18:21:43.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
18:21:43.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/master-2/wal
18:21:43.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/master-2/wal
18:21:43.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/master-2/data/data
18:21:43.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
18:21:43.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
18:21:43.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
18:21:43.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
18:21:43.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
18:21:43.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
18:21:43.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
18:21:43.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
18:21:43.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
18:21:43.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
18:21:43.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
18:21:43.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:43.564242 19479 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.238.188" port: 40453 } attempt: 6
18:21:43.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:43.573169 19479 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.238.188:40453: Network error: Client connection negotiation failed: client connection to 127.18.238.188:40453: connect: Connection refused (error 111)
18:21:43.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:43.608464 19569 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
18:21:43.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:43.610072 19569 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
18:21:43.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:43.610584 19569 kserver.cc:163] Server-wide thread pool size limit: 3276
18:21:43.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:43.684432 19569 rpc_server.cc:307] RPC server started. Bound to: 127.18.238.188:40453
18:21:43.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:43.684513 19663 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.238.188:40453 every 8 connection(s)
18:21:43.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:43.687507 19569 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/master-2/data/info.pb
18:21:43.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:43.692365 19386 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu as pid 19569
18:21:43.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:43.692862 19386 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/master-2/wal/instance
18:21:43.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:43.693774 19664 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
18:21:43.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:43.709000 19664 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.238.190" port: 40173 } has no permanent_uuid. Determining permanent_uuid...
18:21:43.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:43.730625 19664 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.238.189" port: 35025 } has no permanent_uuid. Determining permanent_uuid...
18:21:43.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:43.739647 19664 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.238.188" port: 40453 } has no permanent_uuid. Determining permanent_uuid...
18:21:43.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:43.754724 19664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86bb2b3f4c994cbf9341c7bcb469f41d: Bootstrap starting.
18:21:43.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:43.760972 19664 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86bb2b3f4c994cbf9341c7bcb469f41d: Neither blocks nor log segments found. Creating new log.
18:21:43.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:43.762557 19664 log.cc:826] T 00000000000000000000000000000000 P 86bb2b3f4c994cbf9341c7bcb469f41d: Log is configured to *not* fsync() on all Append() calls
18:21:43.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:43.768298 19664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86bb2b3f4c994cbf9341c7bcb469f41d: No bootstrap required, opened a new log
18:21:43.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:43.785835 19664 raft_consensus.cc:359] T 00000000000000000000000000000000 P 86bb2b3f4c994cbf9341c7bcb469f41d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ede11d6ee67428fb7223a5c9bb41382" member_type: VOTER last_known_addr { host: "127.18.238.190" port: 40173 } } peers { permanent_uuid: "9ffbce142df84608a495ea1fb36df8d1" member_type: VOTER last_known_addr { host: "127.18.238.189" port: 35025 } } peers { permanent_uuid: "86bb2b3f4c994cbf9341c7bcb469f41d" member_type: VOTER last_known_addr { host: "127.18.238.188" port: 40453 } }
18:21:43.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:43.786943 19664 raft_consensus.cc:385] T 00000000000000000000000000000000 P 86bb2b3f4c994cbf9341c7bcb469f41d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
18:21:43.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:43.787194 19664 raft_consensus.cc:740] T 00000000000000000000000000000000 P 86bb2b3f4c994cbf9341c7bcb469f41d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86bb2b3f4c994cbf9341c7bcb469f41d, State: Initialized, Role: FOLLOWER
18:21:43.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:43.788161 19664 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86bb2b3f4c994cbf9341c7bcb469f41d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ede11d6ee67428fb7223a5c9bb41382" member_type: VOTER last_known_addr { host: "127.18.238.190" port: 40173 } } peers { permanent_uuid: "9ffbce142df84608a495ea1fb36df8d1" member_type: VOTER last_known_addr { host: "127.18.238.189" port: 35025 } } peers { permanent_uuid: "86bb2b3f4c994cbf9341c7bcb469f41d" member_type: VOTER last_known_addr { host: "127.18.238.188" port: 40453 } }
18:21:43.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:43.790573 19679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86bb2b3f4c994cbf9341c7bcb469f41d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ede11d6ee67428fb7223a5c9bb41382" member_type: VOTER last_known_addr { host: "127.18.238.190" port: 40173 } } peers { permanent_uuid: "9ffbce142df84608a495ea1fb36df8d1" member_type: VOTER last_known_addr { host: "127.18.238.189" port: 35025 } } peers { permanent_uuid: "86bb2b3f4c994cbf9341c7bcb469f41d" member_type: VOTER last_known_addr { host: "127.18.238.188" port: 40453 } } }
18:21:43.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:43.791378 19679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86bb2b3f4c994cbf9341c7bcb469f41d [sys.catalog]: This master's current role is: FOLLOWER
18:21:43.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:43.791566 19664 sys_catalog.cc:565] T 00000000000000000000000000000000 P 86bb2b3f4c994cbf9341c7bcb469f41d [sys.catalog]: configured and running, proceeding with master startup.
18:21:43.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:43.819571 19690 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 86bb2b3f4c994cbf9341c7bcb469f41d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
18:21:43.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:43.819859 19690 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
18:21:43.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:43.814424 19569 master_runner.cc:187] Error getting master registration for 127.18.238.190:40173: OK, instance_id { permanent_uuid: "3ede11d6ee67428fb7223a5c9bb41382" instance_seqno: 1786472499850856 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
18:21:43.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:43.836485 19569 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
18:21:43.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:43.865550 19613 tablet.cc:2406] T 00000000000000000000000000000000 P 86bb2b3f4c994cbf9341c7bcb469f41d: Can't schedule compaction. Clean time has not been advanced past its initial value.
18:21:43.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:43.876150 19679 raft_consensus.cc:493] T 00000000000000000000000000000000 P 86bb2b3f4c994cbf9341c7bcb469f41d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
18:21:43.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:43.876677 19679 raft_consensus.cc:515] T 00000000000000000000000000000000 P 86bb2b3f4c994cbf9341c7bcb469f41d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ede11d6ee67428fb7223a5c9bb41382" member_type: VOTER last_known_addr { host: "127.18.238.190" port: 40173 } } peers { permanent_uuid: "9ffbce142df84608a495ea1fb36df8d1" member_type: VOTER last_known_addr { host: "127.18.238.189" port: 35025 } } peers { permanent_uuid: "86bb2b3f4c994cbf9341c7bcb469f41d" member_type: VOTER last_known_addr { host: "127.18.238.188" port: 40453 } }
18:21:43.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:43.885084 19679 leader_election.cc:290] T 00000000000000000000000000000000 P 86bb2b3f4c994cbf9341c7bcb469f41d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ede11d6ee67428fb7223a5c9bb41382 (127.18.238.190:40173), 9ffbce142df84608a495ea1fb36df8d1 (127.18.238.189:35025)
18:21:44.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.254323 19567 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.238.188" port: 40453 } attempt: 7
18:21:44.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.283257 19567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ffbce142df84608a495ea1fb36df8d1: Bootstrap starting.
18:21:44.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.291023 19567 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ffbce142df84608a495ea1fb36df8d1: Neither blocks nor log segments found. Creating new log.
18:21:44.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.293422 19567 log.cc:826] T 00000000000000000000000000000000 P 9ffbce142df84608a495ea1fb36df8d1: Log is configured to *not* fsync() on all Append() calls
18:21:44.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.298581 19567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ffbce142df84608a495ea1fb36df8d1: No bootstrap required, opened a new log
18:21:44.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.325440 19567 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9ffbce142df84608a495ea1fb36df8d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ede11d6ee67428fb7223a5c9bb41382" member_type: VOTER last_known_addr { host: "127.18.238.190" port: 40173 } } peers { permanent_uuid: "9ffbce142df84608a495ea1fb36df8d1" member_type: VOTER last_known_addr { host: "127.18.238.189" port: 35025 } } peers { permanent_uuid: "86bb2b3f4c994cbf9341c7bcb469f41d" member_type: VOTER last_known_addr { host: "127.18.238.188" port: 40453 } }
18:21:44.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.326253 19567 raft_consensus.cc:385] T 00000000000000000000000000000000 P 9ffbce142df84608a495ea1fb36df8d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
18:21:44.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.326530 19567 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9ffbce142df84608a495ea1fb36df8d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ffbce142df84608a495ea1fb36df8d1, State: Initialized, Role: FOLLOWER
18:21:44.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.327826 19567 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ffbce142df84608a495ea1fb36df8d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ede11d6ee67428fb7223a5c9bb41382" member_type: VOTER last_known_addr { host: "127.18.238.190" port: 40173 } } peers { permanent_uuid: "9ffbce142df84608a495ea1fb36df8d1" member_type: VOTER last_known_addr { host: "127.18.238.189" port: 35025 } } peers { permanent_uuid: "86bb2b3f4c994cbf9341c7bcb469f41d" member_type: VOTER last_known_addr { host: "127.18.238.188" port: 40453 } }
18:21:44.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.330760 19695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ffbce142df84608a495ea1fb36df8d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ede11d6ee67428fb7223a5c9bb41382" member_type: VOTER last_known_addr { host: "127.18.238.190" port: 40173 } } peers { permanent_uuid: "9ffbce142df84608a495ea1fb36df8d1" member_type: VOTER last_known_addr { host: "127.18.238.189" port: 35025 } } peers { permanent_uuid: "86bb2b3f4c994cbf9341c7bcb469f41d" member_type: VOTER last_known_addr { host: "127.18.238.188" port: 40453 } } }
18:21:44.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.332494 19695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ffbce142df84608a495ea1fb36df8d1 [sys.catalog]: This master's current role is: FOLLOWER
18:21:44.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.332603 19567 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9ffbce142df84608a495ea1fb36df8d1 [sys.catalog]: configured and running, proceeding with master startup.
18:21:44.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.332814 19536 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "86bb2b3f4c994cbf9341c7bcb469f41d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ffbce142df84608a495ea1fb36df8d1" is_pre_election: true
18:21:44.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.333827 19536 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 9ffbce142df84608a495ea1fb36df8d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86bb2b3f4c994cbf9341c7bcb469f41d in term 0.
18:21:44.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.335157 19604 leader_election.cc:304] T 00000000000000000000000000000000 P 86bb2b3f4c994cbf9341c7bcb469f41d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86bb2b3f4c994cbf9341c7bcb469f41d, 9ffbce142df84608a495ea1fb36df8d1; no voters: 
18:21:44.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.336035 19679 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 86bb2b3f4c994cbf9341c7bcb469f41d [term 0 FOLLOWER]: Leader pre-election won for term 1
18:21:44.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.336474 19679 raft_consensus.cc:493] T 00000000000000000000000000000000 P 86bb2b3f4c994cbf9341c7bcb469f41d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
18:21:44.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.336809 19679 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 86bb2b3f4c994cbf9341c7bcb469f41d [term 0 FOLLOWER]: Advancing to term 1
18:21:44.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.343672 19679 raft_consensus.cc:515] T 00000000000000000000000000000000 P 86bb2b3f4c994cbf9341c7bcb469f41d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ede11d6ee67428fb7223a5c9bb41382" member_type: VOTER last_known_addr { host: "127.18.238.190" port: 40173 } } peers { permanent_uuid: "9ffbce142df84608a495ea1fb36df8d1" member_type: VOTER last_known_addr { host: "127.18.238.189" port: 35025 } } peers { permanent_uuid: "86bb2b3f4c994cbf9341c7bcb469f41d" member_type: VOTER last_known_addr { host: "127.18.238.188" port: 40453 } }
18:21:44.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.346674 19536 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "86bb2b3f4c994cbf9341c7bcb469f41d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ffbce142df84608a495ea1fb36df8d1"
18:21:44.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.347185 19536 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9ffbce142df84608a495ea1fb36df8d1 [term 0 FOLLOWER]: Advancing to term 1
18:21:44.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.351027 19679 leader_election.cc:290] T 00000000000000000000000000000000 P 86bb2b3f4c994cbf9341c7bcb469f41d [CANDIDATE]: Term 1 election: Requested vote from peers 3ede11d6ee67428fb7223a5c9bb41382 (127.18.238.190:40173), 9ffbce142df84608a495ea1fb36df8d1 (127.18.238.189:35025)
18:21:44.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.359396 19536 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 9ffbce142df84608a495ea1fb36df8d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86bb2b3f4c994cbf9341c7bcb469f41d in term 1.
18:21:44.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.360357 19604 leader_election.cc:304] T 00000000000000000000000000000000 P 86bb2b3f4c994cbf9341c7bcb469f41d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86bb2b3f4c994cbf9341c7bcb469f41d, 9ffbce142df84608a495ea1fb36df8d1; no voters: 
18:21:44.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.361444 19679 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 86bb2b3f4c994cbf9341c7bcb469f41d [term 1 FOLLOWER]: Leader election won for term 1
18:21:44.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.364959 19679 raft_consensus.cc:697] T 00000000000000000000000000000000 P 86bb2b3f4c994cbf9341c7bcb469f41d [term 1 LEADER]: Becoming Leader. State: Replica: 86bb2b3f4c994cbf9341c7bcb469f41d, State: Running, Role: LEADER
18:21:44.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.366496 19679 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86bb2b3f4c994cbf9341c7bcb469f41d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ede11d6ee67428fb7223a5c9bb41382" member_type: VOTER last_known_addr { host: "127.18.238.190" port: 40173 } } peers { permanent_uuid: "9ffbce142df84608a495ea1fb36df8d1" member_type: VOTER last_known_addr { host: "127.18.238.189" port: 35025 } } peers { permanent_uuid: "86bb2b3f4c994cbf9341c7bcb469f41d" member_type: VOTER last_known_addr { host: "127.18.238.188" port: 40453 } }
18:21:44.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.375927 19702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86bb2b3f4c994cbf9341c7bcb469f41d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86bb2b3f4c994cbf9341c7bcb469f41d. Latest consensus state: current_term: 1 leader_uuid: "86bb2b3f4c994cbf9341c7bcb469f41d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ede11d6ee67428fb7223a5c9bb41382" member_type: VOTER last_known_addr { host: "127.18.238.190" port: 40173 } } peers { permanent_uuid: "9ffbce142df84608a495ea1fb36df8d1" member_type: VOTER last_known_addr { host: "127.18.238.189" port: 35025 } } peers { permanent_uuid: "86bb2b3f4c994cbf9341c7bcb469f41d" member_type: VOTER last_known_addr { host: "127.18.238.188" port: 40453 } } }
18:21:44.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.377184 19702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86bb2b3f4c994cbf9341c7bcb469f41d [sys.catalog]: This master's current role is: LEADER
18:21:44.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.380484 19710 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
18:21:44.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.384325 19709 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
18:21:44.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.392402 19710 catalog_manager.cc:1520] Initializing Kudu cluster ID...
18:21:44.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.411010 19536 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 9ffbce142df84608a495ea1fb36df8d1 [term 1 FOLLOWER]: Refusing update from remote peer 86bb2b3f4c994cbf9341c7bcb469f41d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
18:21:44.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.412863 19679 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 86bb2b3f4c994cbf9341c7bcb469f41d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ffbce142df84608a495ea1fb36df8d1" member_type: VOTER last_known_addr { host: "127.18.238.189" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
18:21:44.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.388501 19482 master_runner.cc:187] Error getting master registration for 127.18.238.190:40173: OK, instance_id { permanent_uuid: "3ede11d6ee67428fb7223a5c9bb41382" instance_seqno: 1786472499850856 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
18:21:44.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.416929 19482 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
18:21:44.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.429188 19695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ffbce142df84608a495ea1fb36df8d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86bb2b3f4c994cbf9341c7bcb469f41d. Latest consensus state: current_term: 1 leader_uuid: "86bb2b3f4c994cbf9341c7bcb469f41d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ede11d6ee67428fb7223a5c9bb41382" member_type: VOTER last_known_addr { host: "127.18.238.190" port: 40173 } } peers { permanent_uuid: "9ffbce142df84608a495ea1fb36df8d1" member_type: VOTER last_known_addr { host: "127.18.238.189" port: 35025 } } peers { permanent_uuid: "86bb2b3f4c994cbf9341c7bcb469f41d" member_type: VOTER last_known_addr { host: "127.18.238.188" port: 40453 } } }
18:21:44.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.431075 19695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ffbce142df84608a495ea1fb36df8d1 [sys.catalog]: This master's current role is: FOLLOWER
18:21:44.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.438791 19695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ffbce142df84608a495ea1fb36df8d1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "86bb2b3f4c994cbf9341c7bcb469f41d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ede11d6ee67428fb7223a5c9bb41382" member_type: VOTER last_known_addr { host: "127.18.238.190" port: 40173 } } peers { permanent_uuid: "9ffbce142df84608a495ea1fb36df8d1" member_type: VOTER last_known_addr { host: "127.18.238.189" port: 35025 } } peers { permanent_uuid: "86bb2b3f4c994cbf9341c7bcb469f41d" member_type: VOTER last_known_addr { host: "127.18.238.188" port: 40453 } } }
18:21:44.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.439677 19695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ffbce142df84608a495ea1fb36df8d1 [sys.catalog]: This master's current role is: FOLLOWER
18:21:44.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.448161 19710 catalog_manager.cc:1383] Generated new cluster ID: f66279a0caf4451c8161a9d8dd97a358
18:21:44.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.448609 19710 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
18:21:44.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.452260 19702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86bb2b3f4c994cbf9341c7bcb469f41d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "86bb2b3f4c994cbf9341c7bcb469f41d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ede11d6ee67428fb7223a5c9bb41382" member_type: VOTER last_known_addr { host: "127.18.238.190" port: 40173 } } peers { permanent_uuid: "9ffbce142df84608a495ea1fb36df8d1" member_type: VOTER last_known_addr { host: "127.18.238.189" port: 35025 } } peers { permanent_uuid: "86bb2b3f4c994cbf9341c7bcb469f41d" member_type: VOTER last_known_addr { host: "127.18.238.188" port: 40453 } } }
18:21:44.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.452924 19702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86bb2b3f4c994cbf9341c7bcb469f41d [sys.catalog]: This master's current role is: LEADER
18:21:44.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.488970 19710 catalog_manager.cc:1406] Generated new certificate authority record
18:21:44.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.490998 19710 catalog_manager.cc:1540] Loading token signing keys...
18:21:44.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.522477 19710 catalog_manager.cc:6092] T 00000000000000000000000000000000 P 86bb2b3f4c994cbf9341c7bcb469f41d: Generated new TSK 0
18:21:44.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.523928 19710 catalog_manager.cc:1550] Initializing in-progress tserver states...
18:21:44.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.541930 19386 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu
18:21:44.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu
18:21:44.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/wal
18:21:44.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/data
18:21:44.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
18:21:44.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
18:21:44.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
18:21:44.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:21:44.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:21:44.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:21:44.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/logs
18:21:44.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/data/info.pb
18:21:44.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:21:44.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
18:21:44.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
18:21:44.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
18:21:44.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
18:21:44.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:21:44.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:21:44.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:21:44.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
18:21:44.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
18:21:44.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.18.238.129:0
18:21:44.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.18.238.129
18:21:44.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.18.238.129
18:21:44.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
18:21:44.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.18.238.190:40173,127.18.238.189:35025,127.18.238.188:40453
18:21:44.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.18.238.148:35877
18:21:44.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:21:44.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:21:44.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
18:21:44.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.606683 19479 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.238.188" port: 40453 } attempt: 7
18:21:44.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.633355 19479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ede11d6ee67428fb7223a5c9bb41382: Bootstrap starting.
18:21:44.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.640926 19479 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ede11d6ee67428fb7223a5c9bb41382: Neither blocks nor log segments found. Creating new log.
18:21:44.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.643433 19479 log.cc:826] T 00000000000000000000000000000000 P 3ede11d6ee67428fb7223a5c9bb41382: Log is configured to *not* fsync() on all Append() calls
18:21:44.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.649179 19479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ede11d6ee67428fb7223a5c9bb41382: No bootstrap required, opened a new log
18:21:44.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.677219 19479 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3ede11d6ee67428fb7223a5c9bb41382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ede11d6ee67428fb7223a5c9bb41382" member_type: VOTER last_known_addr { host: "127.18.238.190" port: 40173 } } peers { permanent_uuid: "9ffbce142df84608a495ea1fb36df8d1" member_type: VOTER last_known_addr { host: "127.18.238.189" port: 35025 } } peers { permanent_uuid: "86bb2b3f4c994cbf9341c7bcb469f41d" member_type: VOTER last_known_addr { host: "127.18.238.188" port: 40453 } }
18:21:44.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.678603 19479 raft_consensus.cc:385] T 00000000000000000000000000000000 P 3ede11d6ee67428fb7223a5c9bb41382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
18:21:44.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.678952 19479 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3ede11d6ee67428fb7223a5c9bb41382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ede11d6ee67428fb7223a5c9bb41382, State: Initialized, Role: FOLLOWER
18:21:44.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.680181 19479 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ede11d6ee67428fb7223a5c9bb41382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ede11d6ee67428fb7223a5c9bb41382" member_type: VOTER last_known_addr { host: "127.18.238.190" port: 40173 } } peers { permanent_uuid: "9ffbce142df84608a495ea1fb36df8d1" member_type: VOTER last_known_addr { host: "127.18.238.189" port: 35025 } } peers { permanent_uuid: "86bb2b3f4c994cbf9341c7bcb469f41d" member_type: VOTER last_known_addr { host: "127.18.238.188" port: 40453 } }
18:21:44.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.688702 19724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ede11d6ee67428fb7223a5c9bb41382 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ede11d6ee67428fb7223a5c9bb41382" member_type: VOTER last_known_addr { host: "127.18.238.190" port: 40173 } } peers { permanent_uuid: "9ffbce142df84608a495ea1fb36df8d1" member_type: VOTER last_known_addr { host: "127.18.238.189" port: 35025 } } peers { permanent_uuid: "86bb2b3f4c994cbf9341c7bcb469f41d" member_type: VOTER last_known_addr { host: "127.18.238.188" port: 40453 } } }
18:21:44.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.692564 19724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ede11d6ee67428fb7223a5c9bb41382 [sys.catalog]: This master's current role is: FOLLOWER
18:21:44.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.692190 19479 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3ede11d6ee67428fb7223a5c9bb41382 [sys.catalog]: configured and running, proceeding with master startup.
18:21:44.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.694281 19451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "86bb2b3f4c994cbf9341c7bcb469f41d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ede11d6ee67428fb7223a5c9bb41382"
18:21:44.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.694052 19450 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3ede11d6ee67428fb7223a5c9bb41382 [term 0 FOLLOWER]: Advancing to term 1
18:21:44.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.696456 19452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "86bb2b3f4c994cbf9341c7bcb469f41d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ede11d6ee67428fb7223a5c9bb41382" is_pre_election: true
18:21:44.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.717473 19450 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 3ede11d6ee67428fb7223a5c9bb41382 [term 1 FOLLOWER]: Refusing update from remote peer 86bb2b3f4c994cbf9341c7bcb469f41d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
18:21:44.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.735611 19679 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 86bb2b3f4c994cbf9341c7bcb469f41d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ede11d6ee67428fb7223a5c9bb41382" member_type: VOTER last_known_addr { host: "127.18.238.190" port: 40173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
18:21:44.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.829887 19724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ede11d6ee67428fb7223a5c9bb41382 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86bb2b3f4c994cbf9341c7bcb469f41d. Latest consensus state: current_term: 1 leader_uuid: "86bb2b3f4c994cbf9341c7bcb469f41d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ede11d6ee67428fb7223a5c9bb41382" member_type: VOTER last_known_addr { host: "127.18.238.190" port: 40173 } } peers { permanent_uuid: "9ffbce142df84608a495ea1fb36df8d1" member_type: VOTER last_known_addr { host: "127.18.238.189" port: 35025 } } peers { permanent_uuid: "86bb2b3f4c994cbf9341c7bcb469f41d" member_type: VOTER last_known_addr { host: "127.18.238.188" port: 40453 } } }
18:21:44.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.831663 19724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ede11d6ee67428fb7223a5c9bb41382 [sys.catalog]: This master's current role is: FOLLOWER
18:21:44.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.832098 19741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ede11d6ee67428fb7223a5c9bb41382 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "86bb2b3f4c994cbf9341c7bcb469f41d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ede11d6ee67428fb7223a5c9bb41382" member_type: VOTER last_known_addr { host: "127.18.238.190" port: 40173 } } peers { permanent_uuid: "9ffbce142df84608a495ea1fb36df8d1" member_type: VOTER last_known_addr { host: "127.18.238.189" port: 35025 } } peers { permanent_uuid: "86bb2b3f4c994cbf9341c7bcb469f41d" member_type: VOTER last_known_addr { host: "127.18.238.188" port: 40453 } } }
18:21:44.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.832859 19741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ede11d6ee67428fb7223a5c9bb41382 [sys.catalog]: This master's current role is: FOLLOWER
18:21:44.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:44.849627 19745 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
18:21:44.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:44.996417 19721 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
18:21:44.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:44.996826 19721 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
18:21:44.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:44.997246 19721 flags.cc:432] Enabled unsafe flag: --never_fsync=true
18:21:45.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:45.029392 19721 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
18:21:45.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:45.030236 19721 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.238.129
18:21:45.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:45.065002 19721 tablet_server_runner.cc:78] Tablet server non-default flags:
18:21:45.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/logs
18:21:45.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:21:45.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
18:21:45.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:21:45.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.18.238.148:35877
18:21:45.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:21:45.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
18:21:45.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/data
18:21:45.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/wal
18:21:45.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:21:45.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:21:45.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.18.238.129:0
18:21:45.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
18:21:45.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:21:45.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:21:45.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/data/info.pb
18:21:45.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.18.238.129
18:21:45.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
18:21:45.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.18.238.190:40173,127.18.238.189:35025,127.18.238.188:40453
18:21:45.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
18:21:45.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:21:45.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
18:21:45.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
18:21:45.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:21:45.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.18.238.129
18:21:45.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
18:21:45.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
18:21:45.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
18:21:45.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision cfdbd367ce7fd9747cb6f1860fc64263d9f9f79a
18:21:45.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
18:21:45.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 11 Aug 2026 18:01:34 UTC on bdcb31816ec0
18:21:45.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 17632
18:21:45.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
18:21:45.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:45.066433 19721 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
18:21:45.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:45.068234 19721 file_cache.cc:504] Constructed file cache file cache with capacity 419430
18:21:45.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:45.081089 19753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:21:45.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:45.253448 19679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86bb2b3f4c994cbf9341c7bcb469f41d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "86bb2b3f4c994cbf9341c7bcb469f41d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ede11d6ee67428fb7223a5c9bb41382" member_type: VOTER last_known_addr { host: "127.18.238.190" port: 40173 } } peers { permanent_uuid: "9ffbce142df84608a495ea1fb36df8d1" member_type: VOTER last_known_addr { host: "127.18.238.189" port: 35025 } } peers { permanent_uuid: "86bb2b3f4c994cbf9341c7bcb469f41d" member_type: VOTER last_known_addr { host: "127.18.238.188" port: 40453 } } }
18:21:45.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:45.255180 19679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86bb2b3f4c994cbf9341c7bcb469f41d [sys.catalog]: This master's current role is: LEADER
18:21:45.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:45.406836 19709 catalog_manager.cc:1295] Loaded cluster ID: f66279a0caf4451c8161a9d8dd97a358
18:21:45.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:45.408650 19709 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 9ffbce142df84608a495ea1fb36df8d1: loading cluster ID for follower catalog manager: success
18:21:45.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:45.436431 19709 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 9ffbce142df84608a495ea1fb36df8d1: acquiring CA information for follower catalog manager: success
18:21:45.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:45.458385 19709 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 9ffbce142df84608a495ea1fb36df8d1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
18:21:45.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:45.893527 19745 catalog_manager.cc:1295] Loaded cluster ID: f66279a0caf4451c8161a9d8dd97a358
18:21:45.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:45.895112 19745 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 3ede11d6ee67428fb7223a5c9bb41382: loading cluster ID for follower catalog manager: success
18:21:45.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:45.910326 19745 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 3ede11d6ee67428fb7223a5c9bb41382: acquiring CA information for follower catalog manager: success
18:21:45.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:45.930586 19745 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 3ede11d6ee67428fb7223a5c9bb41382: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
18:21:47.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:46.483963 19752 debug-util.cc:398] Leaking SignalData structure 0x7b080002ede0 after lost signal to thread 19721
18:21:47.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:47.047313 19756 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1960 milliseconds
18:21:47.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:47.043428 19757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:21:47.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:45.083389 19754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:21:47.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:47.041163 19721 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.959s	user 0.552s	sys 0.945s
18:21:47.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:47.050042 19721 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.968s	user 0.552s	sys 0.946s
18:21:47.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:47.050555 19721 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
18:21:47.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:47.053849 19721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
18:21:47.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:47.065640 19721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
18:21:47.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:47.067816 19721 hybrid_clock.cc:648] HybridClock initialized: now 1786472507067729 us; error 81 us; skew 500 ppm
18:21:47.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:47.069867 19721 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
18:21:47.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:47.098572 19721 webserver.cc:533] Webserver started at http://127.18.238.129:44313/ using document root <none> and password file <none>
18:21:47.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:47.100047 19721 fs_manager.cc:362] Metadata directory not provided
18:21:47.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:47.100541 19721 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
18:21:47.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:47.101295 19721 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
18:21:47.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:47.107776 19721 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/data/instance:
18:21:47.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "708d740e01414324ad249bf1854e9e5e"
18:21:47.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 18:21:47 on dist-test-slave-6nmv"
18:21:47.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:47.109651 19721 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/wal/instance:
18:21:47.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "708d740e01414324ad249bf1854e9e5e"
18:21:47.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 18:21:47 on dist-test-slave-6nmv"
18:21:47.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:47.124987 19721 fs_manager.cc:696] Time spent creating directory manager: real 0.014s	user 0.010s	sys 0.003s
18:21:47.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:47.138741 19769 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
18:21:47.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:47.139905 19721 fs_manager.cc:730] Time spent opening block manager: real 0.010s	user 0.004s	sys 0.006s
18:21:47.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:47.140244 19721 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/data,/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/wal
18:21:47.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "708d740e01414324ad249bf1854e9e5e"
18:21:47.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 18:21:47 on dist-test-slave-6nmv"
18:21:47.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:47.140630 19721 fs_report.cc:389] FS layout report
18:21:47.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
18:21:47.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/wal
18:21:47.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/wal
18:21:47.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/data/data
18:21:47.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
18:21:47.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
18:21:47.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
18:21:47.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
18:21:47.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
18:21:47.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
18:21:47.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
18:21:47.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
18:21:47.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
18:21:47.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
18:21:47.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
18:21:47.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:47.232303 19721 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
18:21:47.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:47.234247 19721 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
18:21:47.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:47.234850 19721 kserver.cc:163] Server-wide thread pool size limit: 3276
18:21:47.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:47.244153 19721 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
18:21:47.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:47.256614 19721 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
18:21:47.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:47.256856 19721 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
18:21:47.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:47.257169 19721 ts_tablet_manager.cc:616] Registered 0 tablets
18:21:47.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:47.257401 19721 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
18:21:47.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:47.498564 19721 rpc_server.cc:307] RPC server started. Bound to: 127.18.238.129:41173
18:21:47.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:47.498667 19910 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.238.129:41173 every 8 connection(s)
18:21:47.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:47.501379 19721 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/data/info.pb
18:21:47.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:47.502525 19386 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu as pid 19721
18:21:47.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:47.502998 19386 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/wal/instance
18:21:47.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:47.516016 19386 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu
18:21:47.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu
18:21:47.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-1/wal
18:21:47.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-1/data
18:21:47.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
18:21:47.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
18:21:47.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
18:21:47.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:21:47.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:21:47.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:21:47.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-1/logs
18:21:47.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-1/data/info.pb
18:21:47.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:21:47.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
18:21:47.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
18:21:47.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
18:21:47.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
18:21:47.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:21:47.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:21:47.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:21:47.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
18:21:47.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
18:21:47.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.18.238.130:0
18:21:47.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.18.238.130
18:21:47.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.18.238.130
18:21:47.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
18:21:47.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.18.238.190:40173,127.18.238.189:35025,127.18.238.188:40453
18:21:47.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.18.238.148:35877
18:21:47.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:21:47.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:21:47.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
18:21:47.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:47.585230 19911 heartbeater.cc:344] Connected to a master server at 127.18.238.189:35025
18:21:47.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:47.585919 19911 heartbeater.cc:461] Registering TS with master...
18:21:47.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:47.587525 19911 heartbeater.cc:507] Master 127.18.238.189:35025 requested a full tablet report, sending...
18:21:47.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:47.591184 19526 ts_manager.cc:194] Registered new tserver with Master: 708d740e01414324ad249bf1854e9e5e (127.18.238.129:41173)
18:21:47.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:47.622375 19912 heartbeater.cc:344] Connected to a master server at 127.18.238.188:40453
18:21:47.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:47.622725 19912 heartbeater.cc:461] Registering TS with master...
18:21:47.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:47.623625 19912 heartbeater.cc:507] Master 127.18.238.188:40453 requested a full tablet report, sending...
18:21:47.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:47.626596 19625 ts_manager.cc:194] Registered new tserver with Master: 708d740e01414324ad249bf1854e9e5e (127.18.238.129:41173)
18:21:47.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:47.629272 19625 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.18.238.129:52191
18:21:47.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:47.633124 19915 heartbeater.cc:344] Connected to a master server at 127.18.238.190:40173
18:21:47.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:47.635192 19915 heartbeater.cc:461] Registering TS with master...
18:21:47.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:47.636979 19915 heartbeater.cc:507] Master 127.18.238.190:40173 requested a full tablet report, sending...
18:21:47.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:47.639824 19441 ts_manager.cc:194] Registered new tserver with Master: 708d740e01414324ad249bf1854e9e5e (127.18.238.129:41173)
18:21:47.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:47.980659 19916 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
18:21:47.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:47.981252 19916 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
18:21:47.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:47.981984 19916 flags.cc:432] Enabled unsafe flag: --never_fsync=true
18:21:48.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:48.014140 19916 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
18:21:48.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:48.015348 19916 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.238.130
18:21:48.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:48.067647 19916 tablet_server_runner.cc:78] Tablet server non-default flags:
18:21:48.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-1/logs
18:21:48.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:21:48.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
18:21:48.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:21:48.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.18.238.148:35877
18:21:48.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:21:48.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
18:21:48.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-1/data
18:21:48.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-1/wal
18:21:48.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:21:48.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:21:48.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.18.238.130:0
18:21:48.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
18:21:48.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:21:48.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:21:48.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-1/data/info.pb
18:21:48.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.18.238.130
18:21:48.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
18:21:48.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.18.238.190:40173,127.18.238.189:35025,127.18.238.188:40453
18:21:48.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
18:21:48.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:21:48.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
18:21:48.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
18:21:48.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:21:48.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.18.238.130
18:21:48.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
18:21:48.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
18:21:48.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
18:21:48.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision cfdbd367ce7fd9747cb6f1860fc64263d9f9f79a
18:21:48.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
18:21:48.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 11 Aug 2026 18:01:34 UTC on bdcb31816ec0
18:21:48.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 17632
18:21:48.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
18:21:48.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:48.069429 19916 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
18:21:48.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:48.071749 19916 file_cache.cc:504] Constructed file cache file cache with capacity 419430
18:21:48.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:48.091764 19931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:21:48.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:48.637244 19912 heartbeater.cc:499] Master 127.18.238.188:40453 was elected leader, sending a full tablet report...
18:21:50.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:49.493171 19930 debug-util.cc:398] Leaking SignalData structure 0x7b080002ede0 after lost signal to thread 19916
18:21:50.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:50.027834 19916 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.933s	user 0.000s	sys 0.002s
18:21:50.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:50.029930 19916 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.936s	user 0.000s	sys 0.003s
18:21:50.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:48.094362 19932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:21:50.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:50.030859 19933 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1934 milliseconds
18:21:50.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:50.031262 19916 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
18:21:50.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:50.031241 19935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:21:50.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:50.036695 19916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
18:21:50.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:50.039685 19916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
18:21:50.041 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:50.041419 19916 hybrid_clock.cc:648] HybridClock initialized: now 1786472510041361 us; error 51 us; skew 500 ppm
18:21:50.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:50.042639 19916 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
18:21:50.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:50.050961 19916 webserver.cc:533] Webserver started at http://127.18.238.130:44475/ using document root <none> and password file <none>
18:21:50.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:50.052217 19916 fs_manager.cc:362] Metadata directory not provided
18:21:50.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:50.052502 19916 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
18:21:50.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:50.053076 19916 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
18:21:50.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:50.059629 19916 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-1/data/instance:
18:21:50.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "24001d6c96764c04999eb5de15b9eba9"
18:21:50.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 18:21:50 on dist-test-slave-6nmv"
18:21:50.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:50.061091 19916 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-1/wal/instance:
18:21:50.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "24001d6c96764c04999eb5de15b9eba9"
18:21:50.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 18:21:50 on dist-test-slave-6nmv"
18:21:50.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:50.070510 19916 fs_manager.cc:696] Time spent creating directory manager: real 0.009s	user 0.010s	sys 0.001s
18:21:50.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:50.077961 19943 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
18:21:50.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:50.079149 19916 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.001s
18:21:50.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:50.079517 19916 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-1/data,/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-1/wal
18:21:50.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "24001d6c96764c04999eb5de15b9eba9"
18:21:50.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 18:21:50 on dist-test-slave-6nmv"
18:21:50.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:50.079885 19916 fs_report.cc:389] FS layout report
18:21:50.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
18:21:50.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-1/wal
18:21:50.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-1/wal
18:21:50.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-1/data/data
18:21:50.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
18:21:50.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
18:21:50.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
18:21:50.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
18:21:50.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
18:21:50.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
18:21:50.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
18:21:50.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
18:21:50.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
18:21:50.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
18:21:50.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
18:21:50.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:50.140555 19916 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
18:21:50.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:50.142266 19916 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
18:21:50.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:50.142773 19916 kserver.cc:163] Server-wide thread pool size limit: 3276
18:21:50.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:50.145586 19916 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
18:21:50.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:50.149789 19916 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
18:21:50.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:50.150038 19916 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.001s
18:21:50.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:50.150343 19916 ts_tablet_manager.cc:616] Registered 0 tablets
18:21:50.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:50.150583 19916 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
18:21:50.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:50.293733 19916 rpc_server.cc:307] RPC server started. Bound to: 127.18.238.130:35737
18:21:50.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:50.293839 20074 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.238.130:35737 every 8 connection(s)
18:21:50.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:50.296478 19916 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-1/data/info.pb
18:21:50.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:50.303478 19386 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu as pid 19916
18:21:50.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:50.304284 19386 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-1/wal/instance
18:21:50.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:50.335204 20075 heartbeater.cc:344] Connected to a master server at 127.18.238.189:35025
18:21:50.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:50.336150 20075 heartbeater.cc:461] Registering TS with master...
18:21:50.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:50.337955 20075 heartbeater.cc:507] Master 127.18.238.189:35025 requested a full tablet report, sending...
18:21:50.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:50.338371 19386 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu
18:21:50.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu
18:21:50.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-2/wal
18:21:50.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-2/data
18:21:50.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
18:21:50.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
18:21:50.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
18:21:50.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:21:50.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:21:50.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:21:50.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-2/logs
18:21:50.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-2/data/info.pb
18:21:50.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:21:50.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
18:21:50.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
18:21:50.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
18:21:50.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
18:21:50.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:21:50.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:21:50.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:21:50.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
18:21:50.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
18:21:50.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.18.238.131:0
18:21:50.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.18.238.131
18:21:50.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.18.238.131
18:21:50.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
18:21:50.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.18.238.190:40173,127.18.238.189:35025,127.18.238.188:40453
18:21:50.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.18.238.148:35877
18:21:50.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:21:50.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:21:50.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
18:21:50.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:50.341572 19526 ts_manager.cc:194] Registered new tserver with Master: 24001d6c96764c04999eb5de15b9eba9 (127.18.238.130:35737)
18:21:50.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:50.345747 20076 heartbeater.cc:344] Connected to a master server at 127.18.238.188:40453
18:21:50.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:50.346096 20076 heartbeater.cc:461] Registering TS with master...
18:21:50.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:50.347046 20076 heartbeater.cc:507] Master 127.18.238.188:40453 requested a full tablet report, sending...
18:21:50.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:50.347849 20077 heartbeater.cc:344] Connected to a master server at 127.18.238.190:40173
18:21:50.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:50.348222 20077 heartbeater.cc:461] Registering TS with master...
18:21:50.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:50.349129 20077 heartbeater.cc:507] Master 127.18.238.190:40173 requested a full tablet report, sending...
18:21:50.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:50.349308 19625 ts_manager.cc:194] Registered new tserver with Master: 24001d6c96764c04999eb5de15b9eba9 (127.18.238.130:35737)
18:21:50.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:50.350809 19625 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.18.238.130:41547
18:21:50.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:50.352347 19441 ts_manager.cc:194] Registered new tserver with Master: 24001d6c96764c04999eb5de15b9eba9 (127.18.238.130:35737)
18:21:50.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:50.721592 20086 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
18:21:50.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:50.722187 20086 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
18:21:50.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:50.722594 20086 flags.cc:432] Enabled unsafe flag: --never_fsync=true
18:21:50.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:50.752754 20086 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
18:21:50.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:50.753552 20086 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.238.131
18:21:50.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:50.788465 20086 tablet_server_runner.cc:78] Tablet server non-default flags:
18:21:50.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-2/logs
18:21:50.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:21:50.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
18:21:50.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:21:50.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.18.238.148:35877
18:21:50.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:21:50.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
18:21:50.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-2/data
18:21:50.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-2/wal
18:21:50.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:21:50.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:21:50.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.18.238.131:0
18:21:50.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
18:21:50.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:21:50.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:21:50.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-2/data/info.pb
18:21:50.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.18.238.131
18:21:50.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
18:21:50.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.18.238.190:40173,127.18.238.189:35025,127.18.238.188:40453
18:21:50.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
18:21:50.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:21:50.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
18:21:50.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
18:21:50.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:21:50.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.18.238.131
18:21:50.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
18:21:50.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
18:21:50.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
18:21:50.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision cfdbd367ce7fd9747cb6f1860fc64263d9f9f79a
18:21:50.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
18:21:50.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 11 Aug 2026 18:01:34 UTC on bdcb31816ec0
18:21:50.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 17632
18:21:50.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
18:21:50.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:50.790136 20086 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
18:21:50.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:50.792356 20086 file_cache.cc:504] Constructed file cache file cache with capacity 419430
18:21:50.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:50.809547 20092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:21:51.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:51.354413 20076 heartbeater.cc:499] Master 127.18.238.188:40453 was elected leader, sending a full tablet report...
18:21:52.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:50.812295 20093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:21:52.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:52.066654 20095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:21:52.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:52.069897 20094 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1255 milliseconds
18:21:52.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:52.069970 20086 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
18:21:52.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:52.071202 20086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
18:21:52.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:52.074318 20086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
18:21:52.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:52.075693 20086 hybrid_clock.cc:648] HybridClock initialized: now 1786472512075663 us; error 46 us; skew 500 ppm
18:21:52.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:52.076485 20086 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
18:21:52.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:52.086441 20086 webserver.cc:533] Webserver started at http://127.18.238.131:34759/ using document root <none> and password file <none>
18:21:52.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:52.087689 20086 fs_manager.cc:362] Metadata directory not provided
18:21:52.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:52.088178 20086 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
18:21:52.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:52.088807 20086 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
18:21:52.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:52.096161 20086 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-2/data/instance:
18:21:52.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "489d187f67e24039b9c5ea35f7d1e11e"
18:21:52.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 18:21:52 on dist-test-slave-6nmv"
18:21:52.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:52.097127 20086 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-2/wal/instance:
18:21:52.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "489d187f67e24039b9c5ea35f7d1e11e"
18:21:52.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 18:21:52 on dist-test-slave-6nmv"
18:21:52.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:52.105211 20086 fs_manager.cc:696] Time spent creating directory manager: real 0.008s	user 0.005s	sys 0.005s
18:21:52.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:52.110833 20102 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
18:21:52.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:52.111860 20086 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.002s
18:21:52.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:52.112210 20086 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-2/data,/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-2/wal
18:21:52.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "489d187f67e24039b9c5ea35f7d1e11e"
18:21:52.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 18:21:52 on dist-test-slave-6nmv"
18:21:52.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:52.112581 20086 fs_report.cc:389] FS layout report
18:21:52.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
18:21:52.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-2/wal
18:21:52.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-2/wal
18:21:52.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-2/data/data
18:21:52.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
18:21:52.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
18:21:52.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
18:21:52.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
18:21:52.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
18:21:52.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
18:21:52.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
18:21:52.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
18:21:52.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
18:21:52.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
18:21:52.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
18:21:52.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:52.202611 20086 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
18:21:52.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:52.204203 20086 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
18:21:52.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:52.204707 20086 kserver.cc:163] Server-wide thread pool size limit: 3276
18:21:52.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:52.207233 20086 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
18:21:52.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:52.211378 20086 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
18:21:52.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:52.211609 20086 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
18:21:52.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:52.211952 20086 ts_tablet_manager.cc:616] Registered 0 tablets
18:21:52.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:52.212149 20086 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
18:21:52.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:52.444036 20086 rpc_server.cc:307] RPC server started. Bound to: 127.18.238.131:46557
18:21:52.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:52.444175 20240 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.238.131:46557 every 8 connection(s)
18:21:52.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:52.447573 20086 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-2/data/info.pb
18:21:52.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:52.448122 19386 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu as pid 20086
18:21:52.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:52.448592 19386 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-2/wal/instance
18:21:52.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:52.513935 20247 heartbeater.cc:344] Connected to a master server at 127.18.238.190:40173
18:21:52.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:52.514480 20247 heartbeater.cc:461] Registering TS with master...
18:21:52.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:52.515861 20247 heartbeater.cc:507] Master 127.18.238.190:40173 requested a full tablet report, sending...
18:21:52.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:52.518383 20242 heartbeater.cc:344] Connected to a master server at 127.18.238.189:35025
18:21:52.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:52.518754 20242 heartbeater.cc:461] Registering TS with master...
18:21:52.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:52.519050 19441 ts_manager.cc:194] Registered new tserver with Master: 489d187f67e24039b9c5ea35f7d1e11e (127.18.238.131:46557)
18:21:52.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:52.519798 20242 heartbeater.cc:507] Master 127.18.238.189:35025 requested a full tablet report, sending...
18:21:52.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:52.523079 19526 ts_manager.cc:194] Registered new tserver with Master: 489d187f67e24039b9c5ea35f7d1e11e (127.18.238.131:46557)
18:21:52.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:52.524253 20243 heartbeater.cc:344] Connected to a master server at 127.18.238.188:40453
18:21:52.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:52.524593 20243 heartbeater.cc:461] Registering TS with master...
18:21:52.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:52.525470 20243 heartbeater.cc:507] Master 127.18.238.188:40453 requested a full tablet report, sending...
18:21:52.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:52.527781 19625 ts_manager.cc:194] Registered new tserver with Master: 489d187f67e24039b9c5ea35f7d1e11e (127.18.238.131:46557)
18:21:52.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:52.529764 19625 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.18.238.131:40733
18:21:52.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:52.539625 19386 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
18:21:52.551 [DEBUG - main] (MiniKuduCluster.java:195) Response: 
18:21:52.552 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_masters {
}

18:21:52.561 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_masters {
  masters {
    id {
      type: MASTER
      index: 0
    }
    bound_rpc_address {
      host: "127.18.238.190"
      port: 40173
    }
    bound_http_address {
      host: "127.18.238.190"
      port: 39133
    }
  }
  masters {
    id {
      type: MASTER
      index: 1
    }
    bound_rpc_address {
      host: "127.18.238.189"
      port: 35025
    }
    bound_http_address {
      host: "127.18.238.189"
      port: 45359
    }
  }
  masters {
    id {
      type: MASTER
      index: 2
    }
    bound_rpc_address {
      host: "127.18.238.188"
      port: 40453
    }
    bound_http_address {
      host: "127.18.238.188"
      port: 40865
    }
  }
}

18:21:52.564 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_tservers {
}

18:21:52.567 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_tservers {
  tservers {
    id {
      type: TSERVER
      index: 0
    }
    bound_rpc_address {
      host: "127.18.238.129"
      port: 41173
    }
    bound_http_address {
      host: "127.18.238.129"
      port: 44313
    }
  }
  tservers {
    id {
      type: TSERVER
      index: 1
    }
    bound_rpc_address {
      host: "127.18.238.130"
      port: 35737
    }
    bound_http_address {
      host: "127.18.238.130"
      port: 44475
    }
  }
  tservers {
    id {
      type: TSERVER
      index: 2
    }
    bound_rpc_address {
      host: "127.18.238.131"
      port: 46557
    }
    bound_http_address {
      host: "127.18.238.131"
      port: 34759
    }
  }
}

18:21:52.567 [INFO - main] (KuduTestHarness.java:157) Creating a new Kudu client...
18:21:52.749 [DEBUG - main] (CompositeStrategy.java:59) Unable to load modern Subject methods, relying only on legacy methods
java.lang.NoSuchMethodException: javax.security.auth.Subject.current()
	at java.base/java.lang.Class.getDeclaredMethod(Class.java:2677)
	at org.apache.kudu.client.internals.ModernStrategy.<init>(ModernStrategy.java:51)
	at org.apache.kudu.client.internals.CompositeStrategy.<init>(CompositeStrategy.java:52)
	at org.apache.kudu.client.internals.CompositeStrategy.<clinit>(CompositeStrategy.java:41)
	at org.apache.kudu.client.internals.SecurityManagerCompatibility.get(SecurityManagerCompatibility.java:43)
	at org.apache.kudu.client.SecurityContext.setupSubject(SecurityContext.java:155)
	at org.apache.kudu.client.SecurityContext.<init>(SecurityContext.java:138)
	at org.apache.kudu.client.AsyncKuduClient.<init>(AsyncKuduClient.java:431)
	at org.apache.kudu.client.AsyncKuduClient$AsyncKuduClientBuilder.build(AsyncKuduClient.java:3109)
	at org.apache.kudu.test.KuduTestHarness.before(KuduTestHarness.java:161)
	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:50)
	at org.apache.kudu.test.junit.RetryRule$RetryStatement.doOneAttempt(RetryRule.java:225)
	at org.apache.kudu.test.junit.RetryRule$RetryStatement.evaluate(RetryRule.java:246)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.junit.runners.Suite.runChild(Suite.java:128)
	at org.junit.runners.Suite.runChild(Suite.java:27)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
	at org.junit.runner.JUnitCore.runMain(JUnitCore.java:77)
	at org.junit.runner.JUnitCore.main(JUnitCore.java:36)
18:21:52.795 [DEBUG - main] (SecurityUtil.java:122) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 
18:21:52.954 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.18.238.189:35025(127.18.238.189:35025)] connecting to peer
18:21:53.008 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.18.238.190:40173(127.18.238.190:40173)] connecting to peer
18:21:53.010 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.18.238.188:40453(127.18.238.188:40453)] connecting to peer
18:21:53.028 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.18.238.190:40173(127.18.238.190:40173)] Successfully connected to peer
18:21:53.028 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.18.238.189:35025(127.18.238.189:35025)] Successfully connected to peer
18:21:53.028 [DEBUG - kudu-nio-2] (Connection.java:757) [peer master-127.18.238.188:40453(127.18.238.188:40453)] Successfully connected to peer
18:21:53.128 [DEBUG - kudu-nio-1] (Negotiator.java:528) SASL mechanism PLAIN chosen for peer 127.18.238.190
18:21:53.128 [DEBUG - kudu-nio-0] (Negotiator.java:528) SASL mechanism PLAIN chosen for peer 127.18.238.189
18:21:53.129 [DEBUG - kudu-nio-2] (Negotiator.java:528) SASL mechanism PLAIN chosen for peer 127.18.238.188
18:21:53.331 [DEBUG - kudu-nio-1] (Negotiator.java:918) Authenticated connection [id: 0x355efe91, L:/127.0.0.1:37346 - R:/127.18.238.190:40173] using SASL/PLAIN
18:21:53.331 [DEBUG - kudu-nio-2] (Negotiator.java:918) Authenticated connection [id: 0x30dec356, L:/127.0.0.1:43768 - R:/127.18.238.188:40453] using SASL/PLAIN
18:21:53.331 [DEBUG - kudu-nio-0] (Negotiator.java:918) Authenticated connection [id: 0x462b4037, L:/127.0.0.1:33700 - R:/127.18.238.189:35025] using SASL/PLAIN
18:21:53.370 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet Kudu Master for table 'Kudu Master' with partition [<start>, <end>)
18:21:53.371 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=CreateTable, tablet=null, attempt=1, TimeoutTracker(timeout=50000, elapsed=457), Traces: [0ms] refreshing cache from master, [26ms] Sub RPC ConnectToMaster: sending RPC to server master-127.18.238.189:35025, [91ms] Sub RPC ConnectToMaster: sending RPC to server master-127.18.238.190:40173, [92ms] Sub RPC ConnectToMaster: sending RPC to server master-127.18.238.188:40453, [427ms] Sub RPC ConnectToMaster: received response from server master-127.18.238.190:40173: OK, [427ms] Sub RPC ConnectToMaster: received response from server master-127.18.238.189:35025: OK, [427ms] Sub RPC ConnectToMaster: received response from server master-127.18.238.188:40453: OK, deferred=null) after lookup
18:21:53.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:53.420281 19625 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43768:
18:21:53.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) name: "org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897"
18:21:53.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) schema {
18:21:53.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
18:21:53.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "key"
18:21:53.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
18:21:53.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: true
18:21:53.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
18:21:53.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
18:21:53.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
18:21:53.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
18:21:53.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
18:21:53.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
18:21:53.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column1_i"
18:21:53.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
18:21:53.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
18:21:53.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
18:21:53.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
18:21:53.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
18:21:53.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
18:21:53.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
18:21:53.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
18:21:53.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column2_i"
18:21:53.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
18:21:53.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
18:21:53.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
18:21:53.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
18:21:53.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
18:21:53.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
18:21:53.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
18:21:53.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
18:21:53.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column3_s"
18:21:53.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: STRING
18:21:53.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
18:21:53.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: true
18:21:53.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     encoding: DICT_ENCODING
18:21:53.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     compression: LZ4
18:21:53.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 4096
18:21:53.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
18:21:53.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
18:21:53.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
18:21:53.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
18:21:53.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column4_b"
18:21:53.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: BOOL
18:21:53.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
18:21:53.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
18:21:53.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
18:21:53.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
18:21:53.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
18:21:53.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
18:21:53.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) }
18:21:53.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) partition_schema {
18:21:53.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   range_schema {
18:21:53.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     columns {
18:21:53.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)       name: "key"
18:21:53.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     }
18:21:53.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
18:21:53.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) }
18:21:53.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:53.425187 19625 catalog_manager.cc:7099] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
18:21:53.507 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:894) Opened table 961bf58e5fcb489dae1c38a19663f109
18:21:53.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:53.533139 20243 heartbeater.cc:499] Master 127.18.238.188:40453 was elected leader, sending a full tablet report...
18:21:53.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:53.548048 20161 tablet_service.cc:1511] Processing CreateTablet for tablet a6f8d542eda4472e9df5c193be9aa15d (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897 [id=961bf58e5fcb489dae1c38a19663f109]), partition=RANGE (key) PARTITION UNBOUNDED
18:21:53.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:53.550308 20161 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6f8d542eda4472e9df5c193be9aa15d. 1 dirs total, 0 dirs full, 0 dirs failed
18:21:53.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:53.561522 20000 tablet_service.cc:1511] Processing CreateTablet for tablet a6f8d542eda4472e9df5c193be9aa15d (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897 [id=961bf58e5fcb489dae1c38a19663f109]), partition=RANGE (key) PARTITION UNBOUNDED
18:21:53.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:53.563977 20000 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6f8d542eda4472e9df5c193be9aa15d. 1 dirs total, 0 dirs full, 0 dirs failed
18:21:53.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:53.573653 19832 tablet_service.cc:1511] Processing CreateTablet for tablet a6f8d542eda4472e9df5c193be9aa15d (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897 [id=961bf58e5fcb489dae1c38a19663f109]), partition=RANGE (key) PARTITION UNBOUNDED
18:21:53.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:53.575536 19832 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6f8d542eda4472e9df5c193be9aa15d. 1 dirs total, 0 dirs full, 0 dirs failed
18:21:53.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:53.594043 20284 tablet_bootstrap.cc:492] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e: Bootstrap starting.
18:21:53.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:53.601485 20285 tablet_bootstrap.cc:492] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9: Bootstrap starting.
18:21:53.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:53.605235 20284 tablet_bootstrap.cc:654] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e: Neither blocks nor log segments found. Creating new log.
18:21:53.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:53.607867 20284 log.cc:826] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e: Log is configured to *not* fsync() on all Append() calls
18:21:53.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:53.610102 20285 tablet_bootstrap.cc:654] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9: Neither blocks nor log segments found. Creating new log.
18:21:53.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:53.613230 20285 log.cc:826] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9: Log is configured to *not* fsync() on all Append() calls
18:21:53.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:53.620635 20286 tablet_bootstrap.cc:492] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e: Bootstrap starting.
18:21:53.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:53.629422 20284 tablet_bootstrap.cc:492] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e: No bootstrap required, opened a new log
18:21:53.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:53.630174 20284 ts_tablet_manager.cc:1403] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e: Time spent bootstrapping tablet: real 0.037s	user 0.020s	sys 0.011s
18:21:53.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:53.633178 20286 tablet_bootstrap.cc:654] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e: Neither blocks nor log segments found. Creating new log.
18:21:53.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:53.636168 20286 log.cc:826] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e: Log is configured to *not* fsync() on all Append() calls
18:21:53.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:53.653966 20285 tablet_bootstrap.cc:492] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9: No bootstrap required, opened a new log
18:21:53.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:53.656008 20285 ts_tablet_manager.cc:1403] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9: Time spent bootstrapping tablet: real 0.055s	user 0.020s	sys 0.010s
18:21:53.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:53.658107 20286 tablet_bootstrap.cc:492] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e: No bootstrap required, opened a new log
18:21:53.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:53.659617 20286 ts_tablet_manager.cc:1403] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e: Time spent bootstrapping tablet: real 0.040s	user 0.017s	sys 0.014s
18:21:53.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:53.668241 20284 raft_consensus.cc:359] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:21:53.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:53.669396 20284 raft_consensus.cc:385] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
18:21:53.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:53.669905 20284 raft_consensus.cc:740] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 489d187f67e24039b9c5ea35f7d1e11e, State: Initialized, Role: FOLLOWER
18:21:53.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:53.671144 20284 consensus_queue.cc:260] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:21:53.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:53.684942 20284 ts_tablet_manager.cc:1434] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e: Time spent starting tablet: real 0.054s	user 0.016s	sys 0.026s
18:21:53.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:53.694496 20286 raft_consensus.cc:359] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:21:53.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:53.695653 20286 raft_consensus.cc:385] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
18:21:53.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:53.695999 20286 raft_consensus.cc:740] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 708d740e01414324ad249bf1854e9e5e, State: Initialized, Role: FOLLOWER
18:21:53.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:53.696950 20286 consensus_queue.cc:260] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:21:53.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:53.697332 20285 raft_consensus.cc:359] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:21:53.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:53.700678 20285 raft_consensus.cc:385] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
18:21:53.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:53.700941 20285 raft_consensus.cc:740] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24001d6c96764c04999eb5de15b9eba9, State: Initialized, Role: FOLLOWER
18:21:53.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:53.701851 20285 consensus_queue.cc:260] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:21:53.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:53.708191 20286 ts_tablet_manager.cc:1434] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e: Time spent starting tablet: real 0.048s	user 0.034s	sys 0.006s
18:21:53.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:53.726565 20285 ts_tablet_manager.cc:1434] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9: Time spent starting tablet: real 0.070s	user 0.030s	sys 0.007s
18:21:53.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:53.742795 20248 tablet.cc:2406] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e: Can't schedule compaction. Clean time has not been advanced past its initial value.
18:21:53.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:53.746417 20293 raft_consensus.cc:493] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
18:21:53.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:53.746997 20293 raft_consensus.cc:515] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:21:53.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:53.750756 20293 leader_election.cc:290] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 24001d6c96764c04999eb5de15b9eba9 (127.18.238.130:35737), 489d187f67e24039b9c5ea35f7d1e11e (127.18.238.131:46557)
18:21:53.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:53.768291 20020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6f8d542eda4472e9df5c193be9aa15d" candidate_uuid: "708d740e01414324ad249bf1854e9e5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24001d6c96764c04999eb5de15b9eba9" is_pre_election: true
18:21:53.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:53.769230 20020 raft_consensus.cc:2468] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 708d740e01414324ad249bf1854e9e5e in term 0.
18:21:53.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:53.770047 20186 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6f8d542eda4472e9df5c193be9aa15d" candidate_uuid: "708d740e01414324ad249bf1854e9e5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "489d187f67e24039b9c5ea35f7d1e11e" is_pre_election: true
18:21:53.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:53.770857 20186 raft_consensus.cc:2468] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 708d740e01414324ad249bf1854e9e5e in term 0.
18:21:53.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:53.771306 19773 leader_election.cc:304] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24001d6c96764c04999eb5de15b9eba9, 708d740e01414324ad249bf1854e9e5e; no voters: 
18:21:53.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:53.772281 20293 raft_consensus.cc:2804] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [term 0 FOLLOWER]: Leader pre-election won for term 1
18:21:53.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:53.772652 20293 raft_consensus.cc:493] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
18:21:53.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:53.772981 20293 raft_consensus.cc:3060] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [term 0 FOLLOWER]: Advancing to term 1
18:21:53.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:53.777467 20293 raft_consensus.cc:515] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:21:53.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:53.779386 20293 leader_election.cc:290] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [CANDIDATE]: Term 1 election: Requested vote from peers 24001d6c96764c04999eb5de15b9eba9 (127.18.238.130:35737), 489d187f67e24039b9c5ea35f7d1e11e (127.18.238.131:46557)
18:21:53.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:53.780386 20186 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6f8d542eda4472e9df5c193be9aa15d" candidate_uuid: "708d740e01414324ad249bf1854e9e5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "489d187f67e24039b9c5ea35f7d1e11e"
18:21:53.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:53.780681 20020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6f8d542eda4472e9df5c193be9aa15d" candidate_uuid: "708d740e01414324ad249bf1854e9e5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24001d6c96764c04999eb5de15b9eba9"
18:21:53.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:53.780843 20186 raft_consensus.cc:3060] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 0 FOLLOWER]: Advancing to term 1
18:21:53.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:53.781119 20020 raft_consensus.cc:3060] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 0 FOLLOWER]: Advancing to term 1
18:21:53.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:53.789402 20186 raft_consensus.cc:2468] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 708d740e01414324ad249bf1854e9e5e in term 1.
18:21:53.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:53.790578 20020 raft_consensus.cc:2468] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 708d740e01414324ad249bf1854e9e5e in term 1.
18:21:53.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:53.790892 19771 leader_election.cc:304] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 489d187f67e24039b9c5ea35f7d1e11e, 708d740e01414324ad249bf1854e9e5e; no voters: 
18:21:53.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:53.792261 20293 raft_consensus.cc:2804] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [term 1 FOLLOWER]: Leader election won for term 1
18:21:53.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:53.795586 20293 raft_consensus.cc:697] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [term 1 LEADER]: Becoming Leader. State: Replica: 708d740e01414324ad249bf1854e9e5e, State: Running, Role: LEADER
18:21:53.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:53.797096 20293 consensus_queue.cc:237] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:21:53.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:53.812971 19625 catalog_manager.cc:5719] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e reported cstate change: term changed from 0 to 1, leader changed from <none> to 708d740e01414324ad249bf1854e9e5e (127.18.238.129). New cstate: current_term: 1 leader_uuid: "708d740e01414324ad249bf1854e9e5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } health_report { overall_health: HEALTHY } } }
18:21:53.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:53.826484 20082 tablet.cc:2406] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9: Can't schedule compaction. Clean time has not been advanced past its initial value.
18:21:53.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:53.888759 19920 tablet.cc:2406] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e: Can't schedule compaction. Clean time has not been advanced past its initial value.
18:21:54.024 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:894) Opened table 961bf58e5fcb489dae1c38a19663f109
18:21:54.055 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:21:54.060 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299995, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737),489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557),708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)[L]]}]
18:21:54.061 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=null, attempt=1, sequence_id=-1, TimeoutTracker(timeout=30000, elapsed=29), Traces: [0ms] refreshing cache from master, [1ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [21ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=null) after lookup
18:21:54.070 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)] connecting to peer
18:21:54.074 [DEBUG - kudu-nio-3] (Connection.java:757) [peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)] Successfully connected to peer
18:21:54.146 [DEBUG - kudu-nio-3] (Negotiator.java:918) Authenticated connection [id: 0xdaf628ab, L:/127.0.0.1:37766 - R:/127.18.238.129:41173] using TOKEN/null
18:21:54.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:54.160733 20020 raft_consensus.cc:1275] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 1 FOLLOWER]: Refusing update from remote peer 708d740e01414324ad249bf1854e9e5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
18:21:54.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:54.160851 20186 raft_consensus.cc:1275] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 1 FOLLOWER]: Refusing update from remote peer 708d740e01414324ad249bf1854e9e5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
18:21:54.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:54.162922 20298 consensus_queue.cc:1048] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
18:21:54.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:54.164227 20293 consensus_queue.cc:1048] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
18:21:54.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:54.197078 20304 mvcc.cc:204] Tried to move back new op lower bound from 7317391417985470464 to 7317391416522780672. Current Snapshot: MvccSnapshot[applied={T|T < 7317391417985470464}]
18:21:54.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:54.208273 20306 mvcc.cc:204] Tried to move back new op lower bound from 7317391417985470464 to 7317391416522780672. Current Snapshot: MvccSnapshot[applied={T|T < 7317391417985470464}]
18:21:54.330 [DEBUG - kudu-nio-3] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 3 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897, tablet=a6f8d542eda4472e9df5c193be9aa15d, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
18:21:54.332 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.18.238.129:41173
18:21:54.333 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

18:21:54.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:54.333923 19386 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu with pid 19721
18:21:54.380 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
18:21:54.386 [DEBUG - kudu-nio-3] (Connection.java:239) [peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)] handling channelInactive
18:21:54.387 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)] cleaning up while in state READY due to: connection closed
18:21:54.387 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173) for tablet a6f8d542eda4472e9df5c193be9aa15d: connection closed
18:21:54.399 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:21:54.400 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737),489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557),708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)[L]]}]
18:21:54.400 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=a6f8d542eda4472e9df5c193be9aa15d, attempt=2, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=19), Traces: [0ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [6ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: connection closed, [6ms] delaying RPC due to: Network error: connection closed, [13ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [18ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=Deferred@673244772(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007f5c7c36b7a8@3c93ebd7 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2e7b8111 -> wakeup thread Time-limited test)) after lookup
18:21:54.401 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)] connecting to peer
18:21:54.403 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:21:54.404 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)] cleaning up while in state CONNECTING due to: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:21:54.404 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173) for tablet a6f8d542eda4472e9df5c193be9aa15d: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:21:54.418 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:21:54.418 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737),489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557),708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)[L]]}]
18:21:54.419 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=a6f8d542eda4472e9df5c193be9aa15d, attempt=4, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=38), Traces: [0ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [6ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: connection closed, [6ms] delaying RPC due to: Network error: connection closed, [13ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [18ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [20ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [23ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [23ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [37ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=Deferred@673244772(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007f5c7c36b7a8@3c93ebd7 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@844822036 after retry RPC@814208371), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2e7b8111 -> wakeup thread Time-limited test -> (continuation of Deferred@844822036 after retry RPC@814208371))) after lookup
18:21:54.419 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)] connecting to peer
18:21:54.421 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:21:54.421 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)] cleaning up while in state CONNECTING due to: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:21:54.422 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173) for tablet a6f8d542eda4472e9df5c193be9aa15d: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:21:54.439 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:21:54.439 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737),489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557),708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)[L]]}]
18:21:54.440 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=a6f8d542eda4472e9df5c193be9aa15d, attempt=6, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=58), Traces: [0ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [6ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: connection closed, [6ms] delaying RPC due to: Network error: connection closed, [13ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [18ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [20ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [23ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [23ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [37ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [38ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [40ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [41ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [53ms] refreshing cache from master, [53ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [57ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=Deferred@673244772(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007f5c7c36b7a8@3c93ebd7 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@844822036 after retry RPC@814208371) -> (continuation of Deferred@201614887 after retry RPC@79653664), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2e7b8111 -> wakeup thread Time-limited test -> (continuation of Deferred@844822036 after retry RPC@814208371) -> (continuation of Deferred@201614887 after retry RPC@79653664))) after lookup
18:21:54.441 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)] connecting to peer
18:21:54.443 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:21:54.443 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)] cleaning up while in state CONNECTING due to: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:21:54.443 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173) for tablet a6f8d542eda4472e9df5c193be9aa15d: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:21:54.478 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:21:54.479 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737),489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557),708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)[L]]}]
18:21:54.479 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=a6f8d542eda4472e9df5c193be9aa15d, attempt=8, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=98), Traces: [0ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [6ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: connection closed, [6ms] delaying RPC due to: Network error: connection closed, [13ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [18ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [20ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [23ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [23ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [37ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [38ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [40ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [41ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [53ms] refreshing cache from master, [53ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [57ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [59ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [62ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [62ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [93ms] refreshing cache from master, [93ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [97ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=Deferred@673244772(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007f5c7c36b7a8@3c93ebd7 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@844822036 after retry RPC@814208371) -> (continuation of Deferred@201614887 after retry RPC@79653664) -> (continuation of Deferred@1278091161 after retry RPC@2060323436), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2e7b8111 -> wakeup thread Time-limited test -> (continuation of Deferred@844822036 after retry RPC@814208371) -> (continuation of Deferred@201614887 after retry RPC@79653664) -> (continuation of Deferred@1278091161 after retry RPC@2060323436))) after lookup
18:21:54.480 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)] connecting to peer
18:21:54.482 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:21:54.482 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)] cleaning up while in state CONNECTING due to: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:21:54.483 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173) for tablet a6f8d542eda4472e9df5c193be9aa15d: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:21:55.019 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:21:55.019 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737),489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557),708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)[L]]}]
18:21:55.019 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=a6f8d542eda4472e9df5c193be9aa15d, attempt=10, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=638), Traces: [0ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [6ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: connection closed, [6ms] delaying RPC due to: Network error: connection closed, [13ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [18ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [20ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [23ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [23ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [37ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [38ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [40ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [41ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [53ms] refreshing cache from master, [53ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [57ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [59ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [62ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [62ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [93ms] refreshing cache from master, [93ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [97ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [99ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [101ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [102ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [633ms] refreshing cache from master, [633ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [637ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=Deferred@673244772(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007f5c7c36b7a8@3c93ebd7 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@844822036 after retry RPC@814208371) -> (continuation of Deferred@201614887 after retry RPC@79653664) -> (continuation of Deferred@1278091161 after retry RPC@2060323436) -> (continuation of Deferred@806757418 after retry RPC@784895111), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2e7b8111 -> wakeup thread Time-limited test -> (continuation of Deferred@844822036 after retry RPC@814208371) -> (continuation of Deferred@201614887 after retry RPC@79653664) -> (continuation of Deferred@1278091161 after retry RPC@2060323436) -> (continuation of Deferred@806757418 after retry RPC@784895111))) after lookup
18:21:55.020 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)] connecting to peer
18:21:55.021 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:21:55.022 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)] cleaning up while in state CONNECTING due to: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:21:55.022 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173) for tablet a6f8d542eda4472e9df5c193be9aa15d: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:21:55.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:55.776513 20323 raft_consensus.cc:493] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 708d740e01414324ad249bf1854e9e5e)
18:21:55.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:55.776933 20323 raft_consensus.cc:515] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:21:55.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:55.779569 20323 leader_election.cc:290] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 489d187f67e24039b9c5ea35f7d1e11e (127.18.238.131:46557), 708d740e01414324ad249bf1854e9e5e (127.18.238.129:41173)
18:21:55.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:55.779845 20324 raft_consensus.cc:493] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 708d740e01414324ad249bf1854e9e5e)
18:21:55.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:55.782014 20324 raft_consensus.cc:515] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:21:55.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:55.784652 19947 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.238.129:41173: connect: Connection refused (error 111)
18:21:55.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:55.789691 19947 leader_election.cc:336] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 708d740e01414324ad249bf1854e9e5e (127.18.238.129:41173): Network error: Client connection negotiation failed: client connection to 127.18.238.129:41173: connect: Connection refused (error 111)
18:21:55.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:55.791419 20324 leader_election.cc:290] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 24001d6c96764c04999eb5de15b9eba9 (127.18.238.130:35737), 708d740e01414324ad249bf1854e9e5e (127.18.238.129:41173)
18:21:55.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:55.792392 20105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.238.129:41173: connect: Connection refused (error 111)
18:21:55.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:55.794319 20186 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6f8d542eda4472e9df5c193be9aa15d" candidate_uuid: "24001d6c96764c04999eb5de15b9eba9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "489d187f67e24039b9c5ea35f7d1e11e" is_pre_election: true
18:21:55.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:55.795243 20186 raft_consensus.cc:2468] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24001d6c96764c04999eb5de15b9eba9 in term 1.
18:21:55.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:55.796636 19944 leader_election.cc:304] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24001d6c96764c04999eb5de15b9eba9, 489d187f67e24039b9c5ea35f7d1e11e; no voters: 708d740e01414324ad249bf1854e9e5e
18:21:55.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:55.797534 20323 raft_consensus.cc:2804] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 1 FOLLOWER]: Leader pre-election won for term 2
18:21:55.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:55.797914 20323 raft_consensus.cc:493] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 708d740e01414324ad249bf1854e9e5e)
18:21:55.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:55.798182 20105 leader_election.cc:336] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 708d740e01414324ad249bf1854e9e5e (127.18.238.129:41173): Network error: Client connection negotiation failed: client connection to 127.18.238.129:41173: connect: Connection refused (error 111)
18:21:55.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:55.798290 20323 raft_consensus.cc:3060] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 1 FOLLOWER]: Advancing to term 2
18:21:55.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:55.800838 20020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6f8d542eda4472e9df5c193be9aa15d" candidate_uuid: "489d187f67e24039b9c5ea35f7d1e11e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "24001d6c96764c04999eb5de15b9eba9" is_pre_election: true
18:21:55.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:55.803351 20323 raft_consensus.cc:515] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:21:55.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:55.804282 20020 raft_consensus.cc:2393] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 489d187f67e24039b9c5ea35f7d1e11e in current term 2: Already voted for candidate 24001d6c96764c04999eb5de15b9eba9 in this term.
18:21:55.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:55.805163 20323 leader_election.cc:290] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [CANDIDATE]: Term 2 election: Requested vote from peers 489d187f67e24039b9c5ea35f7d1e11e (127.18.238.131:46557), 708d740e01414324ad249bf1854e9e5e (127.18.238.129:41173)
18:21:55.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:55.806178 20186 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6f8d542eda4472e9df5c193be9aa15d" candidate_uuid: "24001d6c96764c04999eb5de15b9eba9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "489d187f67e24039b9c5ea35f7d1e11e"
18:21:55.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:55.806581 20186 raft_consensus.cc:3060] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 1 FOLLOWER]: Advancing to term 2
18:21:55.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:55.807533 20105 leader_election.cc:304] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 489d187f67e24039b9c5ea35f7d1e11e; no voters: 24001d6c96764c04999eb5de15b9eba9, 708d740e01414324ad249bf1854e9e5e
18:21:55.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:55.809350 19947 leader_election.cc:336] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 708d740e01414324ad249bf1854e9e5e (127.18.238.129:41173): Network error: Client connection negotiation failed: client connection to 127.18.238.129:41173: connect: Connection refused (error 111)
18:21:55.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:55.811542 20186 raft_consensus.cc:2468] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24001d6c96764c04999eb5de15b9eba9 in term 2.
18:21:55.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:55.811951 20324 raft_consensus.cc:2749] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
18:21:55.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:55.812522 19944 leader_election.cc:304] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24001d6c96764c04999eb5de15b9eba9, 489d187f67e24039b9c5ea35f7d1e11e; no voters: 708d740e01414324ad249bf1854e9e5e
18:21:55.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:55.813099 20323 raft_consensus.cc:2804] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 2 FOLLOWER]: Leader election won for term 2
18:21:55.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:55.814658 20323 raft_consensus.cc:697] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 2 LEADER]: Becoming Leader. State: Replica: 24001d6c96764c04999eb5de15b9eba9, State: Running, Role: LEADER
18:21:55.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:55.815490 20323 consensus_queue.cc:237] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:21:55.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:55.824754 19624 catalog_manager.cc:5719] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 reported cstate change: term changed from 1 to 2, leader changed from 708d740e01414324ad249bf1854e9e5e (127.18.238.129) to 24001d6c96764c04999eb5de15b9eba9 (127.18.238.130). New cstate: current_term: 2 leader_uuid: "24001d6c96764c04999eb5de15b9eba9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } health_report { overall_health: UNKNOWN } } }
18:21:56.159 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:21:56.160 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)[L],489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557),708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)]}]
18:21:56.160 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=a6f8d542eda4472e9df5c193be9aa15d, attempt=12, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=1779), Traces: [0ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [6ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: connection closed, [6ms] delaying RPC due to: Network error: connection closed, [13ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [18ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [20ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [23ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [23ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [37ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [38ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [40ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [41ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [53ms] refreshing cache from master, [53ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [57ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [59ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [62ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [62ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [93ms] refreshing cache from master, [93ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [97ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [99ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [101ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [102ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [633ms] refreshing cache from master, [633ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [637ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [639ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [641ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [641ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [1773ms] refreshing cache from master, [1773ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [1778ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=Deferred@673244772(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007f5c7c36b7a8@3c93ebd7 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@844822036 after retry RPC@814208371) -> (continuation of Deferred@201614887 after retry RPC@79653664) -> (continuation of Deferred@1278091161 after retry RPC@2060323436) -> (continuation of Deferred@806757418 after retry RPC@784895111) -> (continuation of Deferred@1676673820 after retry RPC@162928465), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2e7b8111 -> wakeup thread Time-limited test -> (continuation of Deferred@844822036 after retry RPC@814208371) -> (continuation of Deferred@201614887 after retry RPC@79653664) -> (continuation of Deferred@1278091161 after retry RPC@2060323436) -> (continuation of Deferred@806757418 after retry RPC@784895111) -> (continuation of Deferred@1676673820 after retry RPC@162928465))) after lookup
18:21:56.161 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)] connecting to peer
18:21:56.162 [DEBUG - kudu-nio-1] (Connection.java:757) [peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)] Successfully connected to peer
18:21:56.185 [DEBUG - kudu-nio-1] (Negotiator.java:918) Authenticated connection [id: 0x454d780a, L:/127.0.0.1:50308 - R:/127.18.238.130:35737] using TOKEN/null
18:21:56.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:56.196749 20186 raft_consensus.cc:1275] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 2 FOLLOWER]: Refusing update from remote peer 24001d6c96764c04999eb5de15b9eba9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
18:21:56.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:56.198539 20333 consensus_queue.cc:1048] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
18:21:56.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:56.199390 19947 consensus_peers.cc:597] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 -> Peer 708d740e01414324ad249bf1854e9e5e (127.18.238.129:41173): Couldn't send request to peer 708d740e01414324ad249bf1854e9e5e. Status: Network error: Client connection negotiation failed: client connection to 127.18.238.129:41173: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
18:21:56.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:56.215909 20346 mvcc.cc:204] Tried to move back new op lower bound from 7317391426323828736 to 7317391424787415040. Current Snapshot: MvccSnapshot[applied={T|T < 7317391426323828736}]
18:21:56.253 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.18.238.129:41173
18:21:56.254 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

18:21:56.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:56.256071 19386 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu
18:21:56.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu
18:21:56.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/wal
18:21:56.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/data
18:21:56.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
18:21:56.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
18:21:56.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
18:21:56.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:21:56.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:21:56.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:21:56.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/logs
18:21:56.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/data/info.pb
18:21:56.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:21:56.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
18:21:56.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
18:21:56.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
18:21:56.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
18:21:56.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:21:56.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:21:56.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:21:56.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
18:21:56.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
18:21:56.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.18.238.129:41173
18:21:56.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.18.238.129
18:21:56.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.18.238.190:40173,127.18.238.189:35025,127.18.238.188:40453
18:21:56.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44313
18:21:56.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.18.238.129
18:21:56.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.18.238.148:35877
18:21:56.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:21:56.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:21:56.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
18:21:56.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:56.559651 20354 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
18:21:56.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:56.560083 20354 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
18:21:56.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:56.560463 20354 flags.cc:432] Enabled unsafe flag: --never_fsync=true
18:21:56.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:56.590885 20354 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
18:21:56.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:56.591657 20354 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.238.129
18:21:56.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:56.631120 20354 tablet_server_runner.cc:78] Tablet server non-default flags:
18:21:56.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/logs
18:21:56.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:21:56.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
18:21:56.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:21:56.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.18.238.148:35877
18:21:56.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:21:56.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
18:21:56.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/data
18:21:56.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/wal
18:21:56.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:21:56.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:21:56.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.18.238.129:41173
18:21:56.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
18:21:56.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:21:56.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:21:56.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/data/info.pb
18:21:56.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.18.238.129
18:21:56.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44313
18:21:56.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.18.238.190:40173,127.18.238.189:35025,127.18.238.188:40453
18:21:56.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
18:21:56.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:21:56.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
18:21:56.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
18:21:56.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:21:56.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.18.238.129
18:21:56.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
18:21:56.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
18:21:56.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
18:21:56.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision cfdbd367ce7fd9747cb6f1860fc64263d9f9f79a
18:21:56.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
18:21:56.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 11 Aug 2026 18:01:34 UTC on bdcb31816ec0
18:21:56.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 17632
18:21:56.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
18:21:56.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:56.632432 20354 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
18:21:56.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:56.634284 20354 file_cache.cc:504] Constructed file cache file cache with capacity 419430
18:21:56.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:56.646873 20361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:21:57.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:56.647678 20362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:21:57.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:57.828560 20364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:21:57.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:57.830906 20363 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1179 milliseconds
18:21:57.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:57.830996 20354 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
18:21:57.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:57.832333 20354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
18:21:57.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:57.834640 20354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
18:21:57.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:57.836095 20354 hybrid_clock.cc:648] HybridClock initialized: now 1786472517836042 us; error 54 us; skew 500 ppm
18:21:57.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:57.836890 20354 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
18:21:57.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:57.843324 20354 webserver.cc:533] Webserver started at http://127.18.238.129:44313/ using document root <none> and password file <none>
18:21:57.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:57.844235 20354 fs_manager.cc:362] Metadata directory not provided
18:21:57.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:57.844485 20354 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
18:21:57.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:57.851913 20354 fs_manager.cc:714] Time spent opening directory manager: real 0.005s	user 0.001s	sys 0.004s
18:21:57.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:57.856405 20372 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
18:21:57.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:57.857384 20354 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
18:21:57.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:57.857734 20354 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/data,/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/wal
18:21:57.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "708d740e01414324ad249bf1854e9e5e"
18:21:57.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 18:21:47 on dist-test-slave-6nmv"
18:21:57.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:57.859808 20354 fs_report.cc:389] FS layout report
18:21:57.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
18:21:57.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/wal
18:21:57.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/wal
18:21:57.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/data/data
18:21:57.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
18:21:57.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
18:21:57.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
18:21:57.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
18:21:57.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
18:21:57.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
18:21:57.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
18:21:57.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
18:21:57.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
18:21:57.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
18:21:57.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
18:21:57.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:57.918998 20354 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
18:21:57.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:57.920420 20354 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
18:21:57.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:57.920897 20354 kserver.cc:163] Server-wide thread pool size limit: 3276
18:21:57.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:57.923342 20354 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
18:21:57.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:57.928920 20379 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
18:21:57.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:57.936542 20354 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
18:21:57.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:57.936784 20354 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.009s	user 0.001s	sys 0.000s
18:21:57.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:57.937136 20354 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
18:21:57.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:57.941658 20354 ts_tablet_manager.cc:616] Registered 1 tablets
18:21:57.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:57.941881 20354 ts_tablet_manager.cc:595] Time spent register tablets: real 0.005s	user 0.002s	sys 0.003s
18:21:57.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:57.942222 20379 tablet_bootstrap.cc:492] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e: Bootstrap starting.
18:21:57.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:57.999153 20379 log.cc:826] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e: Log is configured to *not* fsync() on all Append() calls
18:21:58.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:58.030820 20379 tablet_bootstrap.cc:492] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
18:21:58.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:58.031689 20379 tablet_bootstrap.cc:492] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e: Bootstrap complete.
18:21:58.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:58.033519 20379 ts_tablet_manager.cc:1403] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e: Time spent bootstrapping tablet: real 0.092s	user 0.059s	sys 0.027s
18:21:58.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:58.056888 20379 raft_consensus.cc:359] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:21:58.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:58.059996 20379 raft_consensus.cc:740] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 708d740e01414324ad249bf1854e9e5e, State: Initialized, Role: FOLLOWER
18:21:58.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:58.061220 20379 consensus_queue.cc:260] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:21:58.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:58.068030 20379 ts_tablet_manager.cc:1434] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e: Time spent starting tablet: real 0.034s	user 0.018s	sys 0.015s
18:21:58.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:58.125061 20354 rpc_server.cc:307] RPC server started. Bound to: 127.18.238.129:41173
18:21:58.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:58.125291 20507 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.238.129:41173 every 8 connection(s)
18:21:58.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:58.127532 20354 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/data/info.pb
18:21:58.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:58.128850 19386 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu as pid 20354
18:21:58.129 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
18:21:58.164 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 6 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897, tablet=a6f8d542eda4472e9df5c193be9aa15d, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
18:21:58.165 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.18.238.130:35737
18:21:58.165 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

18:21:58.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:58.165968 19386 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu with pid 19916
18:21:58.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:58.183140 20508 heartbeater.cc:344] Connected to a master server at 127.18.238.189:35025
18:21:58.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:58.183576 20508 heartbeater.cc:461] Registering TS with master...
18:21:58.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:58.184005 20510 heartbeater.cc:344] Connected to a master server at 127.18.238.190:40173
18:21:58.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:58.184305 20510 heartbeater.cc:461] Registering TS with master...
18:21:58.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:58.184548 20509 heartbeater.cc:344] Connected to a master server at 127.18.238.188:40453
18:21:58.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:58.184779 20508 heartbeater.cc:507] Master 127.18.238.189:35025 requested a full tablet report, sending...
18:21:58.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:58.185281 20510 heartbeater.cc:507] Master 127.18.238.190:40173 requested a full tablet report, sending...
18:21:58.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:58.184839 20509 heartbeater.cc:461] Registering TS with master...
18:21:58.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:58.187364 20509 heartbeater.cc:507] Master 127.18.238.188:40453 requested a full tablet report, sending...
18:21:58.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:58.189013 19441 ts_manager.cc:194] Re-registered known tserver with Master: 708d740e01414324ad249bf1854e9e5e (127.18.238.129:41173)
18:21:58.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:58.189427 19526 ts_manager.cc:194] Re-registered known tserver with Master: 708d740e01414324ad249bf1854e9e5e (127.18.238.129:41173)
18:21:58.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:58.190692 19624 ts_manager.cc:194] Re-registered known tserver with Master: 708d740e01414324ad249bf1854e9e5e (127.18.238.129:41173)
18:21:58.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:58.193013 19624 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.18.238.129:38459
18:21:58.208 [DEBUG - kudu-nio-1] (Connection.java:239) [peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)] handling channelInactive
18:21:58.208 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
18:21:58.208 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)] cleaning up while in state READY due to: connection closed
18:21:58.209 [DEBUG - Time-limited test] (Connection.java:748) [peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)] connecting to peer
18:21:58.210 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737
18:21:58.210 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)] cleaning up while in state CONNECTING due to: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737
18:21:58.210 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737) for tablet a6f8d542eda4472e9df5c193be9aa15d: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737
18:21:58.218 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:21:58.219 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)[L],489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557),708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)]}]
18:21:58.219 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=a6f8d542eda4472e9df5c193be9aa15d, attempt=2, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=10), Traces: [0ms] sending RPC to server 24001d6c96764c04999eb5de15b9eba9, [2ms] received response from server 24001d6c96764c04999eb5de15b9eba9: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [3ms] delaying RPC due to: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [6ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [10ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=Deferred@1010103845(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007f5c7c36b7a8@32e0b077 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7d61c1a -> wakeup thread Time-limited test)) after lookup
18:21:58.220 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)] connecting to peer
18:21:58.220 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737
18:21:58.220 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)] cleaning up while in state CONNECTING due to: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737
18:21:58.221 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737) for tablet a6f8d542eda4472e9df5c193be9aa15d: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737
18:21:58.238 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:21:58.238 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)[L],489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557),708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)]}]
18:21:58.239 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=a6f8d542eda4472e9df5c193be9aa15d, attempt=4, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=30), Traces: [0ms] sending RPC to server 24001d6c96764c04999eb5de15b9eba9, [2ms] received response from server 24001d6c96764c04999eb5de15b9eba9: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [3ms] delaying RPC due to: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [6ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [10ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [11ms] sending RPC to server 24001d6c96764c04999eb5de15b9eba9, [13ms] received response from server 24001d6c96764c04999eb5de15b9eba9: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [13ms] delaying RPC due to: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [30ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=Deferred@1010103845(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007f5c7c36b7a8@32e0b077 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@192571031 after retry RPC@1023118638), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7d61c1a -> wakeup thread Time-limited test -> (continuation of Deferred@192571031 after retry RPC@1023118638))) after lookup
18:21:58.240 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)] connecting to peer
18:21:58.241 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737
18:21:58.241 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)] cleaning up while in state CONNECTING due to: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737
18:21:58.241 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737) for tablet a6f8d542eda4472e9df5c193be9aa15d: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737
18:21:58.258 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:21:58.259 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)[L],489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557),708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)]}]
18:21:58.259 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=a6f8d542eda4472e9df5c193be9aa15d, attempt=6, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=50), Traces: [0ms] sending RPC to server 24001d6c96764c04999eb5de15b9eba9, [2ms] received response from server 24001d6c96764c04999eb5de15b9eba9: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [3ms] delaying RPC due to: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [6ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [10ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [11ms] sending RPC to server 24001d6c96764c04999eb5de15b9eba9, [13ms] received response from server 24001d6c96764c04999eb5de15b9eba9: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [13ms] delaying RPC due to: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [30ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [31ms] sending RPC to server 24001d6c96764c04999eb5de15b9eba9, [33ms] received response from server 24001d6c96764c04999eb5de15b9eba9: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [33ms] delaying RPC due to: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [46ms] refreshing cache from master, [47ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [50ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=Deferred@1010103845(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007f5c7c36b7a8@32e0b077 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@192571031 after retry RPC@1023118638) -> (continuation of Deferred@1581397241 after retry RPC@687884866), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7d61c1a -> wakeup thread Time-limited test -> (continuation of Deferred@192571031 after retry RPC@1023118638) -> (continuation of Deferred@1581397241 after retry RPC@687884866))) after lookup
18:21:58.260 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)] connecting to peer
18:21:58.261 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737
18:21:58.261 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)] cleaning up while in state CONNECTING due to: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737
18:21:58.261 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737) for tablet a6f8d542eda4472e9df5c193be9aa15d: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737
18:21:58.379 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:21:58.379 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)[L],489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557),708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)]}]
18:21:58.380 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=a6f8d542eda4472e9df5c193be9aa15d, attempt=8, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=171), Traces: [0ms] sending RPC to server 24001d6c96764c04999eb5de15b9eba9, [2ms] received response from server 24001d6c96764c04999eb5de15b9eba9: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [3ms] delaying RPC due to: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [6ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [10ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [11ms] sending RPC to server 24001d6c96764c04999eb5de15b9eba9, [13ms] received response from server 24001d6c96764c04999eb5de15b9eba9: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [13ms] delaying RPC due to: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [30ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [31ms] sending RPC to server 24001d6c96764c04999eb5de15b9eba9, [33ms] received response from server 24001d6c96764c04999eb5de15b9eba9: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [33ms] delaying RPC due to: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [46ms] refreshing cache from master, [47ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [50ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [51ms] sending RPC to server 24001d6c96764c04999eb5de15b9eba9, [53ms] received response from server 24001d6c96764c04999eb5de15b9eba9: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [53ms] delaying RPC due to: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [167ms] refreshing cache from master, [167ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [171ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=Deferred@1010103845(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007f5c7c36b7a8@32e0b077 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@192571031 after retry RPC@1023118638) -> (continuation of Deferred@1581397241 after retry RPC@687884866) -> (continuation of Deferred@366501021 after retry RPC@1042372158), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7d61c1a -> wakeup thread Time-limited test -> (continuation of Deferred@192571031 after retry RPC@1023118638) -> (continuation of Deferred@1581397241 after retry RPC@687884866) -> (continuation of Deferred@366501021 after retry RPC@1042372158))) after lookup
18:21:58.380 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)] connecting to peer
18:21:58.382 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737
18:21:58.382 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)] cleaning up while in state CONNECTING due to: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737
18:21:58.382 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737) for tablet a6f8d542eda4472e9df5c193be9aa15d: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737
18:21:58.658 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:21:58.658 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)[L],489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557),708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)]}]
18:21:58.659 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=a6f8d542eda4472e9df5c193be9aa15d, attempt=10, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=450), Traces: [0ms] sending RPC to server 24001d6c96764c04999eb5de15b9eba9, [2ms] received response from server 24001d6c96764c04999eb5de15b9eba9: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [3ms] delaying RPC due to: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [6ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [10ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [11ms] sending RPC to server 24001d6c96764c04999eb5de15b9eba9, [13ms] received response from server 24001d6c96764c04999eb5de15b9eba9: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [13ms] delaying RPC due to: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [30ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [31ms] sending RPC to server 24001d6c96764c04999eb5de15b9eba9, [33ms] received response from server 24001d6c96764c04999eb5de15b9eba9: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [33ms] delaying RPC due to: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [46ms] refreshing cache from master, [47ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [50ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [51ms] sending RPC to server 24001d6c96764c04999eb5de15b9eba9, [53ms] received response from server 24001d6c96764c04999eb5de15b9eba9: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [53ms] delaying RPC due to: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [167ms] refreshing cache from master, [167ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [171ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [172ms] sending RPC to server 24001d6c96764c04999eb5de15b9eba9, [174ms] received response from server 24001d6c96764c04999eb5de15b9eba9: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [174ms] delaying RPC due to: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [446ms] refreshing cache from master, [446ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [450ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=Deferred@1010103845(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007f5c7c36b7a8@32e0b077 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@192571031 after retry RPC@1023118638) -> (continuation of Deferred@1581397241 after retry RPC@687884866) -> (continuation of Deferred@366501021 after retry RPC@1042372158) -> (continuation of Deferred@1836050143 after retry RPC@1498144234), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7d61c1a -> wakeup thread Time-limited test -> (continuation of Deferred@192571031 after retry RPC@1023118638) -> (continuation of Deferred@1581397241 after retry RPC@687884866) -> (continuation of Deferred@366501021 after retry RPC@1042372158) -> (continuation of Deferred@1836050143 after retry RPC@1498144234))) after lookup
18:21:58.660 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)] connecting to peer
18:21:58.661 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737
18:21:58.661 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)] cleaning up while in state CONNECTING due to: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737
18:21:58.662 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737) for tablet a6f8d542eda4472e9df5c193be9aa15d: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737
18:21:59.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:59.196152 20509 heartbeater.cc:499] Master 127.18.238.188:40453 was elected leader, sending a full tablet report...
18:21:59.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:59.486166 20522 raft_consensus.cc:493] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 24001d6c96764c04999eb5de15b9eba9)
18:21:59.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:59.486588 20522 raft_consensus.cc:515] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:21:59.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:59.488212 20522 leader_election.cc:290] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 24001d6c96764c04999eb5de15b9eba9 (127.18.238.130:35737), 708d740e01414324ad249bf1854e9e5e (127.18.238.129:41173)
18:21:59.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:59.496284 20105 leader_election.cc:336] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 24001d6c96764c04999eb5de15b9eba9 (127.18.238.130:35737): Network error: Client connection negotiation failed: client connection to 127.18.238.130:35737: connect: Connection refused (error 111)
18:21:59.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:59.500475 20454 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6f8d542eda4472e9df5c193be9aa15d" candidate_uuid: "489d187f67e24039b9c5ea35f7d1e11e" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "708d740e01414324ad249bf1854e9e5e" is_pre_election: true
18:21:59.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:59.501206 20454 raft_consensus.cc:2468] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 489d187f67e24039b9c5ea35f7d1e11e in term 1.
18:21:59.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:59.502497 20105 leader_election.cc:304] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 489d187f67e24039b9c5ea35f7d1e11e, 708d740e01414324ad249bf1854e9e5e; no voters: 24001d6c96764c04999eb5de15b9eba9
18:21:59.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:59.503093 20522 raft_consensus.cc:2804] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 2 FOLLOWER]: Leader pre-election won for term 3
18:21:59.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:59.503306 20522 raft_consensus.cc:493] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 24001d6c96764c04999eb5de15b9eba9)
18:21:59.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:59.503507 20522 raft_consensus.cc:3060] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 2 FOLLOWER]: Advancing to term 3
18:21:59.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:59.507664 20522 raft_consensus.cc:515] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:21:59.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:59.508986 20522 leader_election.cc:290] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [CANDIDATE]: Term 3 election: Requested vote from peers 24001d6c96764c04999eb5de15b9eba9 (127.18.238.130:35737), 708d740e01414324ad249bf1854e9e5e (127.18.238.129:41173)
18:21:59.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:59.510435 20454 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6f8d542eda4472e9df5c193be9aa15d" candidate_uuid: "489d187f67e24039b9c5ea35f7d1e11e" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "708d740e01414324ad249bf1854e9e5e"
18:21:59.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:59.510921 20454 raft_consensus.cc:3060] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [term 1 FOLLOWER]: Advancing to term 3
18:21:59.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:21:59.511991 20105 leader_election.cc:336] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 24001d6c96764c04999eb5de15b9eba9 (127.18.238.130:35737): Network error: Client connection negotiation failed: client connection to 127.18.238.130:35737: connect: Connection refused (error 111)
18:21:59.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:59.517138 20454 raft_consensus.cc:2468] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 489d187f67e24039b9c5ea35f7d1e11e in term 3.
18:21:59.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:59.518129 20105 leader_election.cc:304] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 489d187f67e24039b9c5ea35f7d1e11e, 708d740e01414324ad249bf1854e9e5e; no voters: 24001d6c96764c04999eb5de15b9eba9
18:21:59.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:59.518705 20522 raft_consensus.cc:2804] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 3 FOLLOWER]: Leader election won for term 3
18:21:59.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:59.519796 20522 raft_consensus.cc:697] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 3 LEADER]: Becoming Leader. State: Replica: 489d187f67e24039b9c5ea35f7d1e11e, State: Running, Role: LEADER
18:21:59.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:59.520664 20522 consensus_queue.cc:237] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:21:59.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:21:59.528347 19624 catalog_manager.cc:5719] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e reported cstate change: term changed from 2 to 3, leader changed from 24001d6c96764c04999eb5de15b9eba9 (127.18.238.130) to 489d187f67e24039b9c5ea35f7d1e11e (127.18.238.131). New cstate: current_term: 3 leader_uuid: "489d187f67e24039b9c5ea35f7d1e11e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } health_report { overall_health: UNKNOWN } } }
18:22:00.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:00.049572 20454 raft_consensus.cc:1275] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [term 3 FOLLOWER]: Refusing update from remote peer 489d187f67e24039b9c5ea35f7d1e11e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
18:22:00.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:00.051492 20540 consensus_queue.cc:1048] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [LEADER]: Connected to new peer: Peer: permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 4, Time since last communication: 0.000s
18:22:00.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:00.068497 20542 mvcc.cc:204] Tried to move back new op lower bound from 7317391426323828736 to 7317391424787415040. Current Snapshot: MvccSnapshot[applied={T|T < 7317391426323828736}]
18:22:00.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:00.070007 20105 consensus_peers.cc:597] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e -> Peer 24001d6c96764c04999eb5de15b9eba9 (127.18.238.130:35737): Couldn't send request to peer 24001d6c96764c04999eb5de15b9eba9. Status: Network error: Client connection negotiation failed: client connection to 127.18.238.130:35737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
18:22:00.479 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:22:00.479 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737),489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)[L],708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)]}]
18:22:00.480 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=a6f8d542eda4472e9df5c193be9aa15d, attempt=12, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=2271), Traces: [0ms] sending RPC to server 24001d6c96764c04999eb5de15b9eba9, [2ms] received response from server 24001d6c96764c04999eb5de15b9eba9: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [3ms] delaying RPC due to: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [6ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [10ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [11ms] sending RPC to server 24001d6c96764c04999eb5de15b9eba9, [13ms] received response from server 24001d6c96764c04999eb5de15b9eba9: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [13ms] delaying RPC due to: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [30ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [31ms] sending RPC to server 24001d6c96764c04999eb5de15b9eba9, [33ms] received response from server 24001d6c96764c04999eb5de15b9eba9: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [33ms] delaying RPC due to: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [46ms] refreshing cache from master, [47ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [50ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [51ms] sending RPC to server 24001d6c96764c04999eb5de15b9eba9, [53ms] received response from server 24001d6c96764c04999eb5de15b9eba9: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [53ms] delaying RPC due to: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [167ms] refreshing cache from master, [167ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [171ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [172ms] sending RPC to server 24001d6c96764c04999eb5de15b9eba9, [174ms] received response from server 24001d6c96764c04999eb5de15b9eba9: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [174ms] delaying RPC due to: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [446ms] refreshing cache from master, [446ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [450ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [451ms] sending RPC to server 24001d6c96764c04999eb5de15b9eba9, [453ms] received response from server 24001d6c96764c04999eb5de15b9eba9: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [454ms] delaying RPC due to: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [2266ms] refreshing cache from master, [2266ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [2271ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=Deferred@1010103845(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007f5c7c36b7a8@32e0b077 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@192571031 after retry RPC@1023118638) -> (continuation of Deferred@1581397241 after retry RPC@687884866) -> (continuation of Deferred@366501021 after retry RPC@1042372158) -> (continuation of Deferred@1836050143 after retry RPC@1498144234) -> (continuation of Deferred@1384429390 after retry RPC@2118530895), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7d61c1a -> wakeup thread Time-limited test -> (continuation of Deferred@192571031 after retry RPC@1023118638) -> (continuation of Deferred@1581397241 after retry RPC@687884866) -> (continuation of Deferred@366501021 after retry RPC@1042372158) -> (continuation of Deferred@1836050143 after retry RPC@1498144234) -> (continuation of Deferred@1384429390 after retry RPC@2118530895))) after lookup
18:22:00.480 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)] connecting to peer
18:22:00.481 [DEBUG - kudu-nio-0] (Connection.java:757) [peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)] Successfully connected to peer
18:22:00.503 [DEBUG - kudu-nio-0] (Negotiator.java:918) Authenticated connection [id: 0x0103e2cd, L:/127.0.0.1:39550 - R:/127.18.238.131:46557] using TOKEN/null
18:22:00.547 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.18.238.130:35737
18:22:00.548 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

18:22:00.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:00.549809 19386 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu
18:22:00.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu
18:22:00.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-1/wal
18:22:00.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-1/data
18:22:00.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
18:22:00.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
18:22:00.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
18:22:00.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:22:00.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:22:00.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:22:00.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-1/logs
18:22:00.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-1/data/info.pb
18:22:00.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:22:00.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
18:22:00.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
18:22:00.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
18:22:00.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
18:22:00.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:22:00.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:22:00.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:22:00.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
18:22:00.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
18:22:00.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.18.238.130:35737
18:22:00.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.18.238.130
18:22:00.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.18.238.190:40173,127.18.238.189:35025,127.18.238.188:40453
18:22:00.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44475
18:22:00.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.18.238.130
18:22:00.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.18.238.148:35877
18:22:00.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:22:00.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:22:00.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
18:22:00.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:00.857767 20562 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
18:22:00.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:00.858388 20562 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
18:22:00.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:00.859005 20562 flags.cc:432] Enabled unsafe flag: --never_fsync=true
18:22:00.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:00.891199 20562 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
18:22:00.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:00.892182 20562 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.238.130
18:22:00.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:00.927942 20562 tablet_server_runner.cc:78] Tablet server non-default flags:
18:22:00.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-1/logs
18:22:00.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:22:00.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
18:22:00.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:22:00.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.18.238.148:35877
18:22:00.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:22:00.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
18:22:00.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-1/data
18:22:00.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-1/wal
18:22:00.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:22:00.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:22:00.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.18.238.130:35737
18:22:00.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
18:22:00.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:22:00.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:22:00.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-1/data/info.pb
18:22:00.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.18.238.130
18:22:00.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44475
18:22:00.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.18.238.190:40173,127.18.238.189:35025,127.18.238.188:40453
18:22:00.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
18:22:00.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:22:00.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
18:22:00.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
18:22:00.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:22:00.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.18.238.130
18:22:00.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
18:22:00.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
18:22:00.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
18:22:00.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision cfdbd367ce7fd9747cb6f1860fc64263d9f9f79a
18:22:00.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
18:22:00.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 11 Aug 2026 18:01:34 UTC on bdcb31816ec0
18:22:00.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 17632
18:22:00.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
18:22:00.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:00.929739 20562 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
18:22:00.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:00.931346 20562 file_cache.cc:504] Constructed file cache file cache with capacity 419430
18:22:00.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:00.943548 20570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:22:01.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:01.056897 20105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.238.130:35737: connect: Connection refused (error 111) [suppressed 4 similar messages]
18:22:02.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:02.347087 20568 debug-util.cc:398] Leaking SignalData structure 0x7b080002ede0 after lost signal to thread 20562
18:22:02.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:02.393613 20562 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.449s	user 0.484s	sys 0.898s
18:22:02.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:02.393970 20562 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.450s	user 0.484s	sys 0.898s
18:22:02.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:00.944518 20571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:22:02.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:02.396036 20573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:22:02.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:02.399358 20572 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1451 milliseconds
18:22:02.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:02.399441 20562 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
18:22:02.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:02.400738 20562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
18:22:02.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:02.402894 20562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
18:22:02.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:02.404273 20562 hybrid_clock.cc:648] HybridClock initialized: now 1786472522404218 us; error 40 us; skew 500 ppm
18:22:02.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:02.405295 20562 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
18:22:02.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:02.412683 20562 webserver.cc:533] Webserver started at http://127.18.238.130:44475/ using document root <none> and password file <none>
18:22:02.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:02.413921 20562 fs_manager.cc:362] Metadata directory not provided
18:22:02.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:02.414228 20562 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
18:22:02.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:02.421751 20562 fs_manager.cc:714] Time spent opening directory manager: real 0.005s	user 0.005s	sys 0.002s
18:22:02.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:02.426131 20590 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
18:22:02.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:02.427166 20562 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
18:22:02.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:02.427533 20562 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-1/data,/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-1/wal
18:22:02.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "24001d6c96764c04999eb5de15b9eba9"
18:22:02.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 18:21:50 on dist-test-slave-6nmv"
18:22:02.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:02.429563 20562 fs_report.cc:389] FS layout report
18:22:02.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
18:22:02.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-1/wal
18:22:02.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-1/wal
18:22:02.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-1/data/data
18:22:02.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
18:22:02.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
18:22:02.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
18:22:02.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
18:22:02.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
18:22:02.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
18:22:02.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
18:22:02.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
18:22:02.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
18:22:02.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
18:22:02.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
18:22:02.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:02.477972 20562 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
18:22:02.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:02.479775 20562 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
18:22:02.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:02.480357 20562 kserver.cc:163] Server-wide thread pool size limit: 3276
18:22:02.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:02.483178 20562 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
18:22:02.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:02.489002 20602 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
18:22:02.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:02.497527 20562 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
18:22:02.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:02.497752 20562 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.010s	user 0.001s	sys 0.001s
18:22:02.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:02.498117 20562 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
18:22:02.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:02.502558 20562 ts_tablet_manager.cc:616] Registered 1 tablets
18:22:02.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:02.502748 20562 ts_tablet_manager.cc:595] Time spent register tablets: real 0.005s	user 0.004s	sys 0.000s
18:22:02.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:02.503104 20602 tablet_bootstrap.cc:492] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9: Bootstrap starting.
18:22:02.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:02.561015 20602 log.cc:826] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9: Log is configured to *not* fsync() on all Append() calls
18:22:02.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:02.602217 20602 tablet_bootstrap.cc:492] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=6 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
18:22:02.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:02.603282 20602 tablet_bootstrap.cc:492] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9: Bootstrap complete.
18:22:02.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:02.605417 20602 ts_tablet_manager.cc:1403] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9: Time spent bootstrapping tablet: real 0.103s	user 0.070s	sys 0.027s
18:22:02.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:02.640077 20602 raft_consensus.cc:359] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:22:02.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:02.644742 20602 raft_consensus.cc:740] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24001d6c96764c04999eb5de15b9eba9, State: Initialized, Role: FOLLOWER
18:22:02.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:02.646258 20602 consensus_queue.cc:260] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:22:02.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:02.650899 20602 ts_tablet_manager.cc:1434] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9: Time spent starting tablet: real 0.045s	user 0.040s	sys 0.006s
18:22:02.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:02.681027 20562 rpc_server.cc:307] RPC server started. Bound to: 127.18.238.130:35737
18:22:02.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:02.681294 20730 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.238.130:35737 every 8 connection(s)
18:22:02.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:02.684635 20562 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-1/data/info.pb
18:22:02.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:02.694449 19386 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu as pid 20562
18:22:02.695 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
18:22:02.743 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 9 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897, tablet=a6f8d542eda4472e9df5c193be9aa15d, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
18:22:02.743 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.18.238.131:46557
18:22:02.744 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

18:22:02.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:02.744800 19386 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu with pid 20086
18:22:02.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:02.782112 20731 heartbeater.cc:344] Connected to a master server at 127.18.238.189:35025
18:22:02.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:02.782656 20733 heartbeater.cc:344] Connected to a master server at 127.18.238.190:40173
18:22:02.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:02.783139 20733 heartbeater.cc:461] Registering TS with master...
18:22:02.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:02.783145 20732 heartbeater.cc:344] Connected to a master server at 127.18.238.188:40453
18:22:02.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:02.783885 20732 heartbeater.cc:461] Registering TS with master...
18:22:02.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:02.784720 20733 heartbeater.cc:507] Master 127.18.238.190:40173 requested a full tablet report, sending...
18:22:02.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:02.785079 20732 heartbeater.cc:507] Master 127.18.238.188:40453 requested a full tablet report, sending...
18:22:02.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:02.782766 20731 heartbeater.cc:461] Registering TS with master...
18:22:02.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:02.786855 20731 heartbeater.cc:507] Master 127.18.238.189:35025 requested a full tablet report, sending...
18:22:02.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:02.790827 19625 ts_manager.cc:194] Re-registered known tserver with Master: 24001d6c96764c04999eb5de15b9eba9 (127.18.238.130:35737)
18:22:02.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:02.791301 19524 ts_manager.cc:194] Re-registered known tserver with Master: 24001d6c96764c04999eb5de15b9eba9 (127.18.238.130:35737)
18:22:02.794 [DEBUG - kudu-nio-0] (Connection.java:239) [peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)] handling channelInactive
18:22:02.794 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)] cleaning up while in state READY due to: connection closed
18:22:02.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:02.794827 20743 negotiation.cc:336] Failed RPC negotiation. Trace:
18:22:02.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 18:22:02.750929 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.18.238.131:42713 (local address 127.18.238.130:35737)
18:22:02.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 18:22:02.754031 (+  3102us) server_negotiation.cc:207] Beginning negotiation
18:22:02.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 18:22:02.754060 (+    29us) server_negotiation.cc:400] Waiting for connection header
18:22:02.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 18:22:02.793047 (+ 38987us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.238.131:42713: BlockingRecv error: recv got EOF from 127.18.238.131:42713 (error 108)
18:22:02.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":2261,"thread_start_us":11591,"threads_started":1}
18:22:02.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:02.797784 19625 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.18.238.130:41329
18:22:02.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:02.803258 19440 ts_manager.cc:194] Re-registered known tserver with Master: 24001d6c96764c04999eb5de15b9eba9 (127.18.238.130:35737)
18:22:02.805 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
18:22:02.806 [DEBUG - Time-limited test] (Connection.java:748) [peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)] connecting to peer
18:22:02.807 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557
18:22:02.807 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)] cleaning up while in state CONNECTING due to: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557
18:22:02.807 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557) for tablet a6f8d542eda4472e9df5c193be9aa15d: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557
18:22:02.818 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:22:02.819 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737),489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)[L],708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)]}]
18:22:02.819 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=a6f8d542eda4472e9df5c193be9aa15d, attempt=2, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=14), Traces: [0ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [2ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [3ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [13ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=Deferred@395938815(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007f5c7c36b7a8@582a3a0c -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1d409a92 -> wakeup thread Time-limited test)) after lookup
18:22:02.820 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)] connecting to peer
18:22:02.821 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557
18:22:02.821 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)] cleaning up while in state CONNECTING due to: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557
18:22:02.821 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557) for tablet a6f8d542eda4472e9df5c193be9aa15d: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557
18:22:02.837 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:22:02.838 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737),489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)[L],708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)]}]
18:22:02.838 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=a6f8d542eda4472e9df5c193be9aa15d, attempt=4, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=33), Traces: [0ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [2ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [3ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [13ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [14ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [16ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [16ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [32ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=Deferred@395938815(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007f5c7c36b7a8@582a3a0c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1245373796 after retry RPC@60435209), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1d409a92 -> wakeup thread Time-limited test -> (continuation of Deferred@1245373796 after retry RPC@60435209))) after lookup
18:22:02.838 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)] connecting to peer
18:22:02.839 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557
18:22:02.839 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)] cleaning up while in state CONNECTING due to: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557
18:22:02.839 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557) for tablet a6f8d542eda4472e9df5c193be9aa15d: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557
18:22:02.879 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:22:02.880 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737),489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)[L],708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)]}]
18:22:02.880 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=a6f8d542eda4472e9df5c193be9aa15d, attempt=6, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=75), Traces: [0ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [2ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [3ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [13ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [14ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [16ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [16ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [32ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [33ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [34ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [34ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [69ms] refreshing cache from master, [70ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [74ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=Deferred@395938815(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007f5c7c36b7a8@582a3a0c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1245373796 after retry RPC@60435209) -> (continuation of Deferred@1798741109 after retry RPC@967188601), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1d409a92 -> wakeup thread Time-limited test -> (continuation of Deferred@1245373796 after retry RPC@60435209) -> (continuation of Deferred@1798741109 after retry RPC@967188601))) after lookup
18:22:02.881 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)] connecting to peer
18:22:02.882 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557
18:22:02.882 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)] cleaning up while in state CONNECTING due to: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557
18:22:02.882 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557) for tablet a6f8d542eda4472e9df5c193be9aa15d: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557
18:22:03.018 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:22:03.018 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737),489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)[L],708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)]}]
18:22:03.018 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=a6f8d542eda4472e9df5c193be9aa15d, attempt=8, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=213), Traces: [0ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [2ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [3ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [13ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [14ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [16ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [16ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [32ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [33ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [34ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [34ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [69ms] refreshing cache from master, [70ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [74ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [75ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [77ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [77ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [209ms] refreshing cache from master, [209ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [212ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=Deferred@395938815(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007f5c7c36b7a8@582a3a0c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1245373796 after retry RPC@60435209) -> (continuation of Deferred@1798741109 after retry RPC@967188601) -> (continuation of Deferred@1297229073 after retry RPC@1165975850), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1d409a92 -> wakeup thread Time-limited test -> (continuation of Deferred@1245373796 after retry RPC@60435209) -> (continuation of Deferred@1798741109 after retry RPC@967188601) -> (continuation of Deferred@1297229073 after retry RPC@1165975850))) after lookup
18:22:03.019 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)] connecting to peer
18:22:03.020 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557
18:22:03.020 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)] cleaning up while in state CONNECTING due to: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557
18:22:03.020 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557) for tablet a6f8d542eda4472e9df5c193be9aa15d: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557
18:22:03.259 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:22:03.259 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737),489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)[L],708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)]}]
18:22:03.260 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=a6f8d542eda4472e9df5c193be9aa15d, attempt=10, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=454), Traces: [0ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [2ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [3ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [13ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [14ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [16ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [16ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [32ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [33ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [34ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [34ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [69ms] refreshing cache from master, [70ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [74ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [75ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [77ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [77ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [209ms] refreshing cache from master, [209ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [212ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [214ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [215ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [215ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [450ms] refreshing cache from master, [450ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [453ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=Deferred@395938815(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007f5c7c36b7a8@582a3a0c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1245373796 after retry RPC@60435209) -> (continuation of Deferred@1798741109 after retry RPC@967188601) -> (continuation of Deferred@1297229073 after retry RPC@1165975850) -> (continuation of Deferred@1261524199 after retry RPC@1970866842), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1d409a92 -> wakeup thread Time-limited test -> (continuation of Deferred@1245373796 after retry RPC@60435209) -> (continuation of Deferred@1798741109 after retry RPC@967188601) -> (continuation of Deferred@1297229073 after retry RPC@1165975850) -> (continuation of Deferred@1261524199 after retry RPC@1970866842))) after lookup
18:22:03.260 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)] connecting to peer
18:22:03.261 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557
18:22:03.262 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)] cleaning up while in state CONNECTING due to: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557
18:22:03.262 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557) for tablet a6f8d542eda4472e9df5c193be9aa15d: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557
18:22:03.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:03.803930 20732 heartbeater.cc:499] Master 127.18.238.188:40453 was elected leader, sending a full tablet report...
18:22:03.898 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:22:03.899 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737),489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)[L],708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)]}]
18:22:03.899 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=a6f8d542eda4472e9df5c193be9aa15d, attempt=12, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=1094), Traces: [0ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [2ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [3ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [13ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [14ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [16ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [16ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [32ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [33ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [34ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [34ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [69ms] refreshing cache from master, [70ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [74ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [75ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [77ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [77ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [209ms] refreshing cache from master, [209ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [212ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [214ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [215ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [215ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [450ms] refreshing cache from master, [450ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [453ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [455ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [457ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [457ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [1089ms] refreshing cache from master, [1089ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [1093ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=Deferred@395938815(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007f5c7c36b7a8@582a3a0c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1245373796 after retry RPC@60435209) -> (continuation of Deferred@1798741109 after retry RPC@967188601) -> (continuation of Deferred@1297229073 after retry RPC@1165975850) -> (continuation of Deferred@1261524199 after retry RPC@1970866842) -> (continuation of Deferred@64151126 after retry RPC@1052916076), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1d409a92 -> wakeup thread Time-limited test -> (continuation of Deferred@1245373796 after retry RPC@60435209) -> (continuation of Deferred@1798741109 after retry RPC@967188601) -> (continuation of Deferred@1297229073 after retry RPC@1165975850) -> (continuation of Deferred@1261524199 after retry RPC@1970866842) -> (continuation of Deferred@64151126 after retry RPC@1052916076))) after lookup
18:22:03.900 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)] connecting to peer
18:22:03.901 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557
18:22:03.901 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)] cleaning up while in state CONNECTING due to: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557
18:22:03.901 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557) for tablet a6f8d542eda4472e9df5c193be9aa15d: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557
18:22:04.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:04.003538 20747 raft_consensus.cc:493] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
18:22:04.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:04.004112 20747 raft_consensus.cc:515] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:22:04.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:04.006498 20747 leader_election.cc:290] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 489d187f67e24039b9c5ea35f7d1e11e (127.18.238.131:46557), 708d740e01414324ad249bf1854e9e5e (127.18.238.129:41173)
18:22:04.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:04.010283 20591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.238.131:46557: connect: Connection refused (error 111)
18:22:04.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:04.014564 20591 leader_election.cc:336] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 489d187f67e24039b9c5ea35f7d1e11e (127.18.238.131:46557): Network error: Client connection negotiation failed: client connection to 127.18.238.131:46557: connect: Connection refused (error 111)
18:22:04.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:04.017129 20454 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6f8d542eda4472e9df5c193be9aa15d" candidate_uuid: "24001d6c96764c04999eb5de15b9eba9" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "708d740e01414324ad249bf1854e9e5e" is_pre_election: true
18:22:04.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:04.018432 20594 leader_election.cc:304] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24001d6c96764c04999eb5de15b9eba9; no voters: 489d187f67e24039b9c5ea35f7d1e11e, 708d740e01414324ad249bf1854e9e5e
18:22:04.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:04.019039 20747 raft_consensus.cc:2749] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
18:22:04.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:04.183295 20752 raft_consensus.cc:493] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 489d187f67e24039b9c5ea35f7d1e11e)
18:22:04.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:04.183688 20752 raft_consensus.cc:515] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:22:04.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:04.186208 20752 leader_election.cc:290] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 24001d6c96764c04999eb5de15b9eba9 (127.18.238.130:35737), 489d187f67e24039b9c5ea35f7d1e11e (127.18.238.131:46557)
18:22:04.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:04.190394 20373 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.238.131:46557: connect: Connection refused (error 111)
18:22:04.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:04.194720 20373 leader_election.cc:336] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 489d187f67e24039b9c5ea35f7d1e11e (127.18.238.131:46557): Network error: Client connection negotiation failed: client connection to 127.18.238.131:46557: connect: Connection refused (error 111)
18:22:04.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:04.198362 20675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6f8d542eda4472e9df5c193be9aa15d" candidate_uuid: "708d740e01414324ad249bf1854e9e5e" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "24001d6c96764c04999eb5de15b9eba9" is_pre_election: true
18:22:04.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:04.199000 20675 raft_consensus.cc:2468] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 708d740e01414324ad249bf1854e9e5e in term 2.
18:22:04.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:04.200572 20375 leader_election.cc:304] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24001d6c96764c04999eb5de15b9eba9, 708d740e01414324ad249bf1854e9e5e; no voters: 489d187f67e24039b9c5ea35f7d1e11e
18:22:04.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:04.201279 20752 raft_consensus.cc:2804] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [term 3 FOLLOWER]: Leader pre-election won for term 4
18:22:04.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:04.201586 20752 raft_consensus.cc:493] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 489d187f67e24039b9c5ea35f7d1e11e)
18:22:04.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:04.201943 20752 raft_consensus.cc:3060] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [term 3 FOLLOWER]: Advancing to term 4
18:22:04.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:04.206403 20752 raft_consensus.cc:515] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:22:04.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:04.207911 20752 leader_election.cc:290] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [CANDIDATE]: Term 4 election: Requested vote from peers 24001d6c96764c04999eb5de15b9eba9 (127.18.238.130:35737), 489d187f67e24039b9c5ea35f7d1e11e (127.18.238.131:46557)
18:22:04.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:04.208786 20675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6f8d542eda4472e9df5c193be9aa15d" candidate_uuid: "708d740e01414324ad249bf1854e9e5e" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "24001d6c96764c04999eb5de15b9eba9"
18:22:04.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:04.209285 20675 raft_consensus.cc:3060] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 2 FOLLOWER]: Advancing to term 4
18:22:04.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:04.211308 20373 leader_election.cc:336] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 489d187f67e24039b9c5ea35f7d1e11e (127.18.238.131:46557): Network error: Client connection negotiation failed: client connection to 127.18.238.131:46557: connect: Connection refused (error 111)
18:22:04.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:04.215758 20675 raft_consensus.cc:2468] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 708d740e01414324ad249bf1854e9e5e in term 4.
18:22:04.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:04.216606 20375 leader_election.cc:304] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24001d6c96764c04999eb5de15b9eba9, 708d740e01414324ad249bf1854e9e5e; no voters: 489d187f67e24039b9c5ea35f7d1e11e
18:22:04.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:04.217221 20752 raft_consensus.cc:2804] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [term 4 FOLLOWER]: Leader election won for term 4
18:22:04.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:04.218847 20752 raft_consensus.cc:697] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [term 4 LEADER]: Becoming Leader. State: Replica: 708d740e01414324ad249bf1854e9e5e, State: Running, Role: LEADER
18:22:04.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:04.219745 20752 consensus_queue.cc:237] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 12, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:22:04.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:04.227775 19625 catalog_manager.cc:5719] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e reported cstate change: term changed from 3 to 4, leader changed from 489d187f67e24039b9c5ea35f7d1e11e (127.18.238.131) to 708d740e01414324ad249bf1854e9e5e (127.18.238.129). New cstate: current_term: 4 leader_uuid: "708d740e01414324ad249bf1854e9e5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } health_report { overall_health: HEALTHY } } }
18:22:04.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:04.713171 20675 raft_consensus.cc:1275] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 4 FOLLOWER]: Refusing update from remote peer 708d740e01414324ad249bf1854e9e5e: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 4 index: 13. (index mismatch)
18:22:04.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:04.714766 20752 consensus_queue.cc:1048] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 8, Time since last communication: 0.000s
18:22:04.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:04.763055 20373 consensus_peers.cc:597] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e -> Peer 489d187f67e24039b9c5ea35f7d1e11e (127.18.238.131:46557): Couldn't send request to peer 489d187f67e24039b9c5ea35f7d1e11e. Status: Network error: Client connection negotiation failed: client connection to 127.18.238.131:46557: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
18:22:07.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:07.218685 20373 consensus_peers.cc:597] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e -> Peer 489d187f67e24039b9c5ea35f7d1e11e (127.18.238.131:46557): Couldn't send request to peer 489d187f67e24039b9c5ea35f7d1e11e. Status: Network error: Client connection negotiation failed: client connection to 127.18.238.131:46557: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
18:22:07.738 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:22:07.739 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737),489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557),708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)[L]]}]
18:22:07.739 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=a6f8d542eda4472e9df5c193be9aa15d, attempt=14, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=4934), Traces: [0ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [2ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [3ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [13ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [14ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [16ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [16ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [32ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [33ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [34ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [34ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [69ms] refreshing cache from master, [70ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [74ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [75ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [77ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [77ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [209ms] refreshing cache from master, [209ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [212ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [214ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [215ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [215ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [450ms] refreshing cache from master, [450ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [453ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [455ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [457ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [457ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [1089ms] refreshing cache from master, [1089ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [1093ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [1094ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [1096ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [1096ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [4929ms] refreshing cache from master, [4929ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [4933ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=Deferred@395938815(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007f5c7c36b7a8@582a3a0c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1245373796 after retry RPC@60435209) -> (continuation of Deferred@1798741109 after retry RPC@967188601) -> (continuation of Deferred@1297229073 after retry RPC@1165975850) -> (continuation of Deferred@1261524199 after retry RPC@1970866842) -> (continuation of Deferred@64151126 after retry RPC@1052916076) -> (continuation of Deferred@534583826 after retry RPC@996827860), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1d409a92 -> wakeup thread Time-limited test -> (continuation of Deferred@1245373796 after retry RPC@60435209) -> (continuation of Deferred@1798741109 after retry RPC@967188601) -> (continuation of Deferred@1297229073 after retry RPC@1165975850) -> (continuation of Deferred@1261524199 after retry RPC@1970866842) -> (continuation of Deferred@64151126 after retry RPC@1052916076) -> (continuation of Deferred@534583826 after retry RPC@996827860))) after lookup
18:22:07.740 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)] connecting to peer
18:22:07.741 [DEBUG - kudu-nio-0] (Connection.java:757) [peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)] Successfully connected to peer
18:22:07.763 [DEBUG - kudu-nio-0] (Negotiator.java:918) Authenticated connection [id: 0x0e5364fb, L:/127.0.0.1:59136 - R:/127.18.238.129:41173] using TOKEN/null
18:22:07.816 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.18.238.131:46557
18:22:07.816 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

18:22:07.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:07.817684 19386 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu
18:22:07.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu
18:22:07.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-2/wal
18:22:07.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-2/data
18:22:07.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
18:22:07.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
18:22:07.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
18:22:07.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:22:07.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:22:07.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:22:07.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-2/logs
18:22:07.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-2/data/info.pb
18:22:07.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:22:07.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
18:22:07.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
18:22:07.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
18:22:07.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
18:22:07.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:22:07.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:22:07.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:22:07.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
18:22:07.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
18:22:07.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.18.238.131:46557
18:22:07.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.18.238.131
18:22:07.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.18.238.190:40173,127.18.238.189:35025,127.18.238.188:40453
18:22:07.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=34759
18:22:07.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.18.238.131
18:22:07.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.18.238.148:35877
18:22:07.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:22:07.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:22:07.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
18:22:08.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:08.158411 20805 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
18:22:08.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:08.159024 20805 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
18:22:08.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:08.159832 20805 flags.cc:432] Enabled unsafe flag: --never_fsync=true
18:22:08.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:08.194685 20805 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
18:22:08.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:08.195822 20805 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.238.131
18:22:08.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:08.230871 20805 tablet_server_runner.cc:78] Tablet server non-default flags:
18:22:08.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-2/logs
18:22:08.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:22:08.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
18:22:08.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:22:08.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.18.238.148:35877
18:22:08.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:22:08.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
18:22:08.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-2/data
18:22:08.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-2/wal
18:22:08.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:22:08.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:22:08.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.18.238.131:46557
18:22:08.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
18:22:08.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:22:08.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:22:08.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-2/data/info.pb
18:22:08.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.18.238.131
18:22:08.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=34759
18:22:08.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.18.238.190:40173,127.18.238.189:35025,127.18.238.188:40453
18:22:08.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
18:22:08.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:22:08.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
18:22:08.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
18:22:08.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:22:08.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.18.238.131
18:22:08.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
18:22:08.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
18:22:08.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
18:22:08.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision cfdbd367ce7fd9747cb6f1860fc64263d9f9f79a
18:22:08.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
18:22:08.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 11 Aug 2026 18:01:34 UTC on bdcb31816ec0
18:22:08.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 17632
18:22:08.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
18:22:08.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:08.232183 20805 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
18:22:08.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:08.233704 20805 file_cache.cc:504] Constructed file cache file cache with capacity 419430
18:22:08.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:08.246246 20814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:22:08.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:08.897439 19561 debug-util.cc:398] Leaking SignalData structure 0x7b0800070080 after lost signal to thread 19488
18:22:08.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:08.898511 19561 debug-util.cc:398] Leaking SignalData structure 0x7b08000a9060 after lost signal to thread 19566
18:22:09.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:08.249862 20817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:22:09.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:08.247651 20815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:22:09.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:09.507109 20816 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
18:22:09.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:09.507262 20805 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
18:22:09.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:09.508796 20805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
18:22:09.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:09.511730 20805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
18:22:09.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:09.513217 20805 hybrid_clock.cc:648] HybridClock initialized: now 1786472529513165 us; error 50 us; skew 500 ppm
18:22:09.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:09.514482 20805 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
18:22:09.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:09.522154 20805 webserver.cc:533] Webserver started at http://127.18.238.131:34759/ using document root <none> and password file <none>
18:22:09.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:09.523420 20805 fs_manager.cc:362] Metadata directory not provided
18:22:09.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:09.523738 20805 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
18:22:09.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:09.534670 20805 fs_manager.cc:714] Time spent opening directory manager: real 0.007s	user 0.006s	sys 0.001s
18:22:09.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:09.540424 20830 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
18:22:09.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:09.541716 20805 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.004s	sys 0.002s
18:22:09.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:09.542114 20805 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-2/data,/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-2/wal
18:22:09.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "489d187f67e24039b9c5ea35f7d1e11e"
18:22:09.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 18:21:52 on dist-test-slave-6nmv"
18:22:09.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:09.544683 20805 fs_report.cc:389] FS layout report
18:22:09.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
18:22:09.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-2/wal
18:22:09.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-2/wal
18:22:09.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-2/data/data
18:22:09.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
18:22:09.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
18:22:09.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
18:22:09.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
18:22:09.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
18:22:09.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
18:22:09.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
18:22:09.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
18:22:09.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
18:22:09.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
18:22:09.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
18:22:09.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:09.572729 20373 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.238.131:46557: connect: Connection refused (error 111) [suppressed 11 similar messages]
18:22:09.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:09.575982 20373 consensus_peers.cc:597] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e -> Peer 489d187f67e24039b9c5ea35f7d1e11e (127.18.238.131:46557): Couldn't send request to peer 489d187f67e24039b9c5ea35f7d1e11e. Status: Network error: Client connection negotiation failed: client connection to 127.18.238.131:46557: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
18:22:09.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:09.613673 20805 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
18:22:09.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:09.615295 20805 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
18:22:09.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:09.615703 20805 kserver.cc:163] Server-wide thread pool size limit: 3276
18:22:09.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:09.618975 20805 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
18:22:09.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:09.626019 20838 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
18:22:09.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:09.637569 20805 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
18:22:09.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:09.637776 20805 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.014s	user 0.002s	sys 0.001s
18:22:09.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:09.638072 20805 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
18:22:09.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:09.643055 20805 ts_tablet_manager.cc:616] Registered 1 tablets
18:22:09.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:09.643280 20805 ts_tablet_manager.cc:595] Time spent register tablets: real 0.005s	user 0.004s	sys 0.000s
18:22:09.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:09.643621 20838 tablet_bootstrap.cc:492] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e: Bootstrap starting.
18:22:09.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:09.721048 20838 log.cc:826] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e: Log is configured to *not* fsync() on all Append() calls
18:22:09.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:09.799872 20838 tablet_bootstrap.cc:492] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=9 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
18:22:09.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:09.800750 20838 tablet_bootstrap.cc:492] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e: Bootstrap complete.
18:22:09.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:09.802479 20838 ts_tablet_manager.cc:1403] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e: Time spent bootstrapping tablet: real 0.159s	user 0.090s	sys 0.043s
18:22:09.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:09.825552 20838 raft_consensus.cc:359] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:22:09.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:09.829025 20838 raft_consensus.cc:740] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 489d187f67e24039b9c5ea35f7d1e11e, State: Initialized, Role: FOLLOWER
18:22:09.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:09.830638 20838 consensus_queue.cc:260] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 3.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:22:09.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:09.836337 20838 ts_tablet_manager.cc:1434] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e: Time spent starting tablet: real 0.034s	user 0.028s	sys 0.004s
18:22:09.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:09.924438 20805 rpc_server.cc:307] RPC server started. Bound to: 127.18.238.131:46557
18:22:09.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:09.924672 20966 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.238.131:46557 every 8 connection(s)
18:22:09.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:09.929468 20805 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-2/data/info.pb
18:22:09.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:09.937976 19386 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu as pid 20805
18:22:09.939 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
18:22:09.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:09.987027 20969 heartbeater.cc:344] Connected to a master server at 127.18.238.190:40173
18:22:09.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:09.987550 20969 heartbeater.cc:461] Registering TS with master...
18:22:09.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:09.989120 20969 heartbeater.cc:507] Master 127.18.238.190:40173 requested a full tablet report, sending...
18:22:09.994 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 12 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897, tablet=a6f8d542eda4472e9df5c193be9aa15d, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
18:22:09.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:09.994462 19440 ts_manager.cc:194] Re-registered known tserver with Master: 489d187f67e24039b9c5ea35f7d1e11e (127.18.238.131:46557)
18:22:10.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:10.000319 20968 heartbeater.cc:344] Connected to a master server at 127.18.238.188:40453
18:22:10.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:10.000674 20968 heartbeater.cc:461] Registering TS with master...
18:22:10.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:10.001710 20968 heartbeater.cc:507] Master 127.18.238.188:40453 requested a full tablet report, sending...
18:22:10.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:10.007476 19625 ts_manager.cc:194] Re-registered known tserver with Master: 489d187f67e24039b9c5ea35f7d1e11e (127.18.238.131:46557)
18:22:10.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:10.010412 19625 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.18.238.131:47165
18:22:10.012 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.18.238.129:41173
18:22:10.012 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

18:22:10.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:10.013334 19386 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu with pid 20354
18:22:10.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:10.016409 20967 heartbeater.cc:344] Connected to a master server at 127.18.238.189:35025
18:22:10.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:10.016755 20967 heartbeater.cc:461] Registering TS with master...
18:22:10.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:10.017784 20967 heartbeater.cc:507] Master 127.18.238.189:35025 requested a full tablet report, sending...
18:22:10.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:10.021497 19524 ts_manager.cc:194] Re-registered known tserver with Master: 489d187f67e24039b9c5ea35f7d1e11e (127.18.238.131:46557)
18:22:10.059 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
18:22:10.059 [DEBUG - kudu-nio-0] (Connection.java:239) [peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)] handling channelInactive
18:22:10.060 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)] cleaning up while in state READY due to: connection closed
18:22:10.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:10.060495 20978 negotiation.cc:336] Failed RPC negotiation. Trace:
18:22:10.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 18:22:09.984838 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.18.238.129:53185 (local address 127.18.238.131:46557)
18:22:10.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 18:22:10.002550 (+ 17712us) server_negotiation.cc:207] Beginning negotiation
18:22:10.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 18:22:10.002584 (+    34us) server_negotiation.cc:400] Waiting for connection header
18:22:10.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 18:22:10.002640 (+    56us) server_negotiation.cc:408] Connection header received
18:22:10.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 18:22:10.002849 (+   209us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
18:22:10.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 18:22:10.002887 (+    38us) server_negotiation.cc:462] Received NEGOTIATE request from client
18:22:10.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 18:22:10.003384 (+   497us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
18:22:10.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 18:22:10.003614 (+   230us) server_negotiation.cc:227] Negotiated authn=SASL
18:22:10.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 18:22:10.008305 (+  4691us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
18:22:10.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 18:22:10.010710 (+  2405us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
18:22:10.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 18:22:10.016464 (+  5754us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
18:22:10.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 18:22:10.017150 (+   686us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
18:22:10.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 18:22:10.058407 (+ 41257us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.238.129:53185: BlockingRecv error: recv got EOF from 127.18.238.129:53185 (error 108)
18:22:10.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":16830,"thread_start_us":16390,"threads_started":1}
18:22:10.075 [DEBUG - pool-2-thread-1] (Connection.java:748) [peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)] connecting to peer
18:22:10.076 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:22:10.076 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)] cleaning up while in state CONNECTING due to: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:22:10.077 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173) for tablet a6f8d542eda4472e9df5c193be9aa15d: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:22:10.098 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:22:10.099 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737),489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557),708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)[L]]}]
18:22:10.099 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=a6f8d542eda4472e9df5c193be9aa15d, attempt=3, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=39), Traces: [0ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [1ms] delaying RPC due to: Illegal state: connection is terminated, [15ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [18ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [18ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [39ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=Deferred@22224042(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007f5c7c36b7a8@69c7a296 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@11218be -> wakeup thread Time-limited test)) after lookup
18:22:10.100 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)] connecting to peer
18:22:10.101 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:22:10.101 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)] cleaning up while in state CONNECTING due to: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:22:10.101 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173) for tablet a6f8d542eda4472e9df5c193be9aa15d: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:22:10.118 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:22:10.119 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737),489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557),708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)[L]]}]
18:22:10.119 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=a6f8d542eda4472e9df5c193be9aa15d, attempt=5, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=59), Traces: [0ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [1ms] delaying RPC due to: Illegal state: connection is terminated, [15ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [18ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [18ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [39ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [40ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [42ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [42ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [55ms] refreshing cache from master, [56ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [59ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=Deferred@22224042(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007f5c7c36b7a8@69c7a296 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@691704250 after retry RPC@31819182), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@11218be -> wakeup thread Time-limited test -> (continuation of Deferred@691704250 after retry RPC@31819182))) after lookup
18:22:10.120 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)] connecting to peer
18:22:10.121 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:22:10.121 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)] cleaning up while in state CONNECTING due to: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:22:10.121 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173) for tablet a6f8d542eda4472e9df5c193be9aa15d: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:22:10.158 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:22:10.159 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737),489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557),708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)[L]]}]
18:22:10.159 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=a6f8d542eda4472e9df5c193be9aa15d, attempt=7, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=99), Traces: [0ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [1ms] delaying RPC due to: Illegal state: connection is terminated, [15ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [18ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [18ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [39ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [40ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [42ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [42ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [55ms] refreshing cache from master, [56ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [59ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [60ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [62ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [62ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [95ms] refreshing cache from master, [95ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [99ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=Deferred@22224042(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007f5c7c36b7a8@69c7a296 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@691704250 after retry RPC@31819182) -> (continuation of Deferred@1548072096 after retry RPC@1906348659), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@11218be -> wakeup thread Time-limited test -> (continuation of Deferred@691704250 after retry RPC@31819182) -> (continuation of Deferred@1548072096 after retry RPC@1906348659))) after lookup
18:22:10.160 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)] connecting to peer
18:22:10.161 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:22:10.161 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)] cleaning up while in state CONNECTING due to: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:22:10.161 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173) for tablet a6f8d542eda4472e9df5c193be9aa15d: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:22:10.338 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:22:10.338 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737),489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557),708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)[L]]}]
18:22:10.339 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=a6f8d542eda4472e9df5c193be9aa15d, attempt=9, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=279), Traces: [0ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [1ms] delaying RPC due to: Illegal state: connection is terminated, [15ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [18ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [18ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [39ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [40ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [42ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [42ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [55ms] refreshing cache from master, [56ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [59ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [60ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [62ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [62ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [95ms] refreshing cache from master, [95ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [99ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [100ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [102ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [102ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [275ms] refreshing cache from master, [275ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [279ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=Deferred@22224042(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007f5c7c36b7a8@69c7a296 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@691704250 after retry RPC@31819182) -> (continuation of Deferred@1548072096 after retry RPC@1906348659) -> (continuation of Deferred@1457658011 after retry RPC@589572926), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@11218be -> wakeup thread Time-limited test -> (continuation of Deferred@691704250 after retry RPC@31819182) -> (continuation of Deferred@1548072096 after retry RPC@1906348659) -> (continuation of Deferred@1457658011 after retry RPC@589572926))) after lookup
18:22:10.339 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)] connecting to peer
18:22:10.341 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:22:10.341 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)] cleaning up while in state CONNECTING due to: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:22:10.341 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173) for tablet a6f8d542eda4472e9df5c193be9aa15d: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:22:11.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:11.021023 20968 heartbeater.cc:499] Master 127.18.238.188:40453 was elected leader, sending a full tablet report...
18:22:11.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:11.185792 20981 raft_consensus.cc:493] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
18:22:11.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:11.186262 20981 raft_consensus.cc:515] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:22:11.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:11.188746 20981 leader_election.cc:290] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 24001d6c96764c04999eb5de15b9eba9 (127.18.238.130:35737), 708d740e01414324ad249bf1854e9e5e (127.18.238.129:41173)
18:22:11.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:11.193431 20833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.238.129:41173: connect: Connection refused (error 111)
18:22:11.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:11.196591 20987 raft_consensus.cc:493] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 708d740e01414324ad249bf1854e9e5e)
18:22:11.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:11.198201 20833 leader_election.cc:336] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 708d740e01414324ad249bf1854e9e5e (127.18.238.129:41173): Network error: Client connection negotiation failed: client connection to 127.18.238.129:41173: connect: Connection refused (error 111)
18:22:11.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:11.197691 20987 raft_consensus.cc:515] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:22:11.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:11.202109 20987 leader_election.cc:290] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 489d187f67e24039b9c5ea35f7d1e11e (127.18.238.131:46557), 708d740e01414324ad249bf1854e9e5e (127.18.238.129:41173)
18:22:11.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:11.205336 20675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6f8d542eda4472e9df5c193be9aa15d" candidate_uuid: "489d187f67e24039b9c5ea35f7d1e11e" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "24001d6c96764c04999eb5de15b9eba9" is_pre_election: true
18:22:11.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:11.206460 20675 raft_consensus.cc:2393] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 489d187f67e24039b9c5ea35f7d1e11e in current term 4: Already voted for candidate 708d740e01414324ad249bf1854e9e5e in this term.
18:22:11.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:11.206555 20594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.238.129:41173: connect: Connection refused (error 111)
18:22:11.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:11.208827 20833 leader_election.cc:304] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 489d187f67e24039b9c5ea35f7d1e11e; no voters: 24001d6c96764c04999eb5de15b9eba9, 708d740e01414324ad249bf1854e9e5e
18:22:11.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:11.209708 20981 raft_consensus.cc:3060] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 3 FOLLOWER]: Advancing to term 4
18:22:11.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:11.211587 20594 leader_election.cc:336] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 708d740e01414324ad249bf1854e9e5e (127.18.238.129:41173): Network error: Client connection negotiation failed: client connection to 127.18.238.129:41173: connect: Connection refused (error 111)
18:22:11.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:11.214564 20911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6f8d542eda4472e9df5c193be9aa15d" candidate_uuid: "24001d6c96764c04999eb5de15b9eba9" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "489d187f67e24039b9c5ea35f7d1e11e" is_pre_election: true
18:22:11.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:11.217034 20981 raft_consensus.cc:2749] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
18:22:11.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:11.217643 20911 raft_consensus.cc:2468] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24001d6c96764c04999eb5de15b9eba9 in term 4.
18:22:11.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:11.222512 20591 leader_election.cc:304] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24001d6c96764c04999eb5de15b9eba9, 489d187f67e24039b9c5ea35f7d1e11e; no voters: 708d740e01414324ad249bf1854e9e5e
18:22:11.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:11.223210 20987 raft_consensus.cc:2804] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 4 FOLLOWER]: Leader pre-election won for term 5
18:22:11.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:11.223489 20987 raft_consensus.cc:493] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 708d740e01414324ad249bf1854e9e5e)
18:22:11.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:11.223691 20987 raft_consensus.cc:3060] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 4 FOLLOWER]: Advancing to term 5
18:22:11.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:11.228222 20987 raft_consensus.cc:515] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:22:11.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:11.229745 20987 leader_election.cc:290] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [CANDIDATE]: Term 5 election: Requested vote from peers 489d187f67e24039b9c5ea35f7d1e11e (127.18.238.131:46557), 708d740e01414324ad249bf1854e9e5e (127.18.238.129:41173)
18:22:11.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:11.230448 20911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6f8d542eda4472e9df5c193be9aa15d" candidate_uuid: "24001d6c96764c04999eb5de15b9eba9" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "489d187f67e24039b9c5ea35f7d1e11e"
18:22:11.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:11.230844 20911 raft_consensus.cc:3060] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 4 FOLLOWER]: Advancing to term 5
18:22:11.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:11.234431 20594 leader_election.cc:336] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 708d740e01414324ad249bf1854e9e5e (127.18.238.129:41173): Network error: Client connection negotiation failed: client connection to 127.18.238.129:41173: connect: Connection refused (error 111)
18:22:11.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:11.236320 20911 raft_consensus.cc:2468] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24001d6c96764c04999eb5de15b9eba9 in term 5.
18:22:11.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:11.237099 20591 leader_election.cc:304] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24001d6c96764c04999eb5de15b9eba9, 489d187f67e24039b9c5ea35f7d1e11e; no voters: 708d740e01414324ad249bf1854e9e5e
18:22:11.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:11.237712 20987 raft_consensus.cc:2804] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 5 FOLLOWER]: Leader election won for term 5
18:22:11.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:11.238888 20987 raft_consensus.cc:697] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 5 LEADER]: Becoming Leader. State: Replica: 24001d6c96764c04999eb5de15b9eba9, State: Running, Role: LEADER
18:22:11.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:11.239903 20987 consensus_queue.cc:237] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 4.16, Last appended by leader: 16, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:22:11.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:11.249864 19625 catalog_manager.cc:5719] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 reported cstate change: term changed from 4 to 5, leader changed from 708d740e01414324ad249bf1854e9e5e (127.18.238.129) to 24001d6c96764c04999eb5de15b9eba9 (127.18.238.130). New cstate: current_term: 5 leader_uuid: "24001d6c96764c04999eb5de15b9eba9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } health_report { overall_health: UNKNOWN } } }
18:22:11.378 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:22:11.378 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)[L],489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557),708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)]}]
18:22:11.379 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=a6f8d542eda4472e9df5c193be9aa15d, attempt=11, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=1319), Traces: [0ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [1ms] delaying RPC due to: Illegal state: connection is terminated, [15ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [18ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [18ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [39ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [40ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [42ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [42ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [55ms] refreshing cache from master, [56ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [59ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [60ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [62ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [62ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [95ms] refreshing cache from master, [95ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [99ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [100ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [102ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [102ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [275ms] refreshing cache from master, [275ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [279ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [280ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [282ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [282ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [1315ms] refreshing cache from master, [1315ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [1319ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=Deferred@22224042(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007f5c7c36b7a8@69c7a296 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@691704250 after retry RPC@31819182) -> (continuation of Deferred@1548072096 after retry RPC@1906348659) -> (continuation of Deferred@1457658011 after retry RPC@589572926) -> (continuation of Deferred@65951643 after retry RPC@17948931), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@11218be -> wakeup thread Time-limited test -> (continuation of Deferred@691704250 after retry RPC@31819182) -> (continuation of Deferred@1548072096 after retry RPC@1906348659) -> (continuation of Deferred@1457658011 after retry RPC@589572926) -> (continuation of Deferred@65951643 after retry RPC@17948931))) after lookup
18:22:11.380 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)] connecting to peer
18:22:11.381 [DEBUG - kudu-nio-6] (Connection.java:757) [peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)] Successfully connected to peer
18:22:11.404 [DEBUG - kudu-nio-6] (Negotiator.java:918) Authenticated connection [id: 0x57685d3a, L:/127.0.0.1:33680 - R:/127.18.238.130:35737] using TOKEN/null
18:22:11.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:11.414772 20911 raft_consensus.cc:1275] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 5 FOLLOWER]: Refusing update from remote peer 24001d6c96764c04999eb5de15b9eba9: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 5 index: 18. (index mismatch)
18:22:11.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:11.415895 20594 consensus_peers.cc:597] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 -> Peer 708d740e01414324ad249bf1854e9e5e (127.18.238.129:41173): Couldn't send request to peer 708d740e01414324ad249bf1854e9e5e. Status: Network error: Client connection negotiation failed: client connection to 127.18.238.129:41173: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
18:22:11.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:11.416456 20993 consensus_queue.cc:1048] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 12, Time since last communication: 0.000s
18:22:11.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:11.439731 21009 mvcc.cc:204] Tried to move back new op lower bound from 7317391488657068032 to 7317391487967379456. Current Snapshot: MvccSnapshot[applied={T|T < 7317391488657068032}]
18:22:11.487 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.18.238.129:41173
18:22:11.487 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

18:22:11.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:11.488847 19386 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu
18:22:11.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu
18:22:11.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/wal
18:22:11.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/data
18:22:11.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
18:22:11.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
18:22:11.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
18:22:11.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:22:11.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:22:11.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:22:11.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/logs
18:22:11.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/data/info.pb
18:22:11.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:22:11.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
18:22:11.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
18:22:11.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
18:22:11.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
18:22:11.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:22:11.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:22:11.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:22:11.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
18:22:11.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
18:22:11.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.18.238.129:41173
18:22:11.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.18.238.129
18:22:11.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.18.238.190:40173,127.18.238.189:35025,127.18.238.188:40453
18:22:11.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44313
18:22:11.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.18.238.129
18:22:11.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.18.238.148:35877
18:22:11.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:22:11.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:22:11.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
18:22:11.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:11.799106 21017 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
18:22:11.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:11.799649 21017 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
18:22:11.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:11.800139 21017 flags.cc:432] Enabled unsafe flag: --never_fsync=true
18:22:11.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:11.837004 21017 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
18:22:11.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:11.838315 21017 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.238.129
18:22:11.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:11.879760 21017 tablet_server_runner.cc:78] Tablet server non-default flags:
18:22:11.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/logs
18:22:11.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:22:11.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
18:22:11.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:22:11.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.18.238.148:35877
18:22:11.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:22:11.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
18:22:11.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/data
18:22:11.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/wal
18:22:11.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:22:11.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:22:11.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.18.238.129:41173
18:22:11.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
18:22:11.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:22:11.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:22:11.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/data/info.pb
18:22:11.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.18.238.129
18:22:11.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44313
18:22:11.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.18.238.190:40173,127.18.238.189:35025,127.18.238.188:40453
18:22:11.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
18:22:11.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:22:11.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
18:22:11.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
18:22:11.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:22:11.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.18.238.129
18:22:11.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
18:22:11.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
18:22:11.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
18:22:11.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision cfdbd367ce7fd9747cb6f1860fc64263d9f9f79a
18:22:11.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
18:22:11.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 11 Aug 2026 18:01:34 UTC on bdcb31816ec0
18:22:11.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 17632
18:22:11.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
18:22:11.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:11.881542 21017 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
18:22:11.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:11.883484 21017 file_cache.cc:504] Constructed file cache file cache with capacity 419430
18:22:11.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:11.896912 21025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:22:13.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:11.898021 21026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:22:13.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:13.133163 21017 thread.cc:641] GCE (cloud detector) Time spent creating pthread: real 1.237s	user 0.399s	sys 0.834s
18:22:13.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:13.133559 21017 thread.cc:608] GCE (cloud detector) Time spent starting thread: real 1.238s	user 0.399s	sys 0.834s
18:22:13.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:13.136838 21017 server_base.cc:1061] running on GCE node
18:22:13.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:13.137540 21037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:22:13.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:13.139168 21017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
18:22:13.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:13.142136 21017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
18:22:13.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:13.143608 21017 hybrid_clock.cc:648] HybridClock initialized: now 1786472533143478 us; error 110 us; skew 500 ppm
18:22:13.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:13.144886 21017 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
18:22:13.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:13.154871 21017 webserver.cc:533] Webserver started at http://127.18.238.129:44313/ using document root <none> and password file <none>
18:22:13.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:13.156349 21017 fs_manager.cc:362] Metadata directory not provided
18:22:13.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:13.156689 21017 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
18:22:13.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:13.168627 21017 fs_manager.cc:714] Time spent opening directory manager: real 0.007s	user 0.001s	sys 0.009s
18:22:13.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:13.175417 21044 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
18:22:13.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:13.176990 21017 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.002s	sys 0.003s
18:22:13.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:13.177436 21017 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/data,/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/wal
18:22:13.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "708d740e01414324ad249bf1854e9e5e"
18:22:13.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 18:21:47 on dist-test-slave-6nmv"
18:22:13.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:13.180518 21017 fs_report.cc:389] FS layout report
18:22:13.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
18:22:13.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/wal
18:22:13.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/wal
18:22:13.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/data/data
18:22:13.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
18:22:13.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
18:22:13.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
18:22:13.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
18:22:13.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
18:22:13.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
18:22:13.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
18:22:13.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
18:22:13.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
18:22:13.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
18:22:13.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
18:22:13.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:13.259018 21017 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
18:22:13.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:13.260682 21017 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
18:22:13.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:13.261117 21017 kserver.cc:163] Server-wide thread pool size limit: 3276
18:22:13.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:13.263852 21017 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
18:22:13.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:13.269450 21051 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
18:22:13.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:13.278250 21017 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
18:22:13.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:13.278512 21017 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.010s	user 0.001s	sys 0.001s
18:22:13.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:13.278863 21017 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
18:22:13.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:13.283640 21017 ts_tablet_manager.cc:616] Registered 1 tablets
18:22:13.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:13.283864 21017 ts_tablet_manager.cc:595] Time spent register tablets: real 0.005s	user 0.004s	sys 0.000s
18:22:13.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:13.284130 21051 tablet_bootstrap.cc:492] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e: Bootstrap starting.
18:22:13.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:13.350530 21051 log.cc:826] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e: Log is configured to *not* fsync() on all Append() calls
18:22:13.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:13.430136 21051 tablet_bootstrap.cc:492] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e: Bootstrap replayed 1/1 log segments. Stats: ops{read=16 overwritten=0 applied=16 ignored=0} inserts{seen=12 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
18:22:13.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:13.431424 21051 tablet_bootstrap.cc:492] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e: Bootstrap complete.
18:22:13.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:13.433609 21051 ts_tablet_manager.cc:1403] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e: Time spent bootstrapping tablet: real 0.150s	user 0.109s	sys 0.036s
18:22:13.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:13.454521 21017 rpc_server.cc:307] RPC server started. Bound to: 127.18.238.129:41173
18:22:13.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:13.454700 21173 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.238.129:41173 every 8 connection(s)
18:22:13.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:13.458137 21017 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/data/info.pb
18:22:13.468 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:13.461584 21051 raft_consensus.cc:359] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:22:13.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:13.468015 21051 raft_consensus.cc:740] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 708d740e01414324ad249bf1854e9e5e, State: Initialized, Role: FOLLOWER
18:22:13.469 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
18:22:13.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:13.468354 19386 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu as pid 21017
18:22:13.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:13.469571 21051 consensus_queue.cc:260] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 4.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:22:13.520 [DEBUG - kudu-nio-6] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 15 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897, tablet=a6f8d542eda4472e9df5c193be9aa15d, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
18:22:13.520 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.18.238.130:35737
18:22:13.521 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

18:22:13.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:13.521719 19386 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu with pid 20562
18:22:13.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:13.527266 21051 ts_tablet_manager.cc:1434] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e: Time spent starting tablet: real 0.093s	user 0.039s	sys 0.017s
18:22:13.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:13.531992 21174 heartbeater.cc:344] Connected to a master server at 127.18.238.189:35025
18:22:13.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:13.532485 21174 heartbeater.cc:461] Registering TS with master...
18:22:13.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:13.533860 21174 heartbeater.cc:507] Master 127.18.238.189:35025 requested a full tablet report, sending...
18:22:13.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:13.534144 21175 heartbeater.cc:344] Connected to a master server at 127.18.238.188:40453
18:22:13.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:13.534514 21175 heartbeater.cc:461] Registering TS with master...
18:22:13.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:13.535391 21175 heartbeater.cc:507] Master 127.18.238.188:40453 requested a full tablet report, sending...
18:22:13.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:13.536922 21177 heartbeater.cc:344] Connected to a master server at 127.18.238.190:40173
18:22:13.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:13.537236 21177 heartbeater.cc:461] Registering TS with master...
18:22:13.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:13.538071 21177 heartbeater.cc:507] Master 127.18.238.190:40173 requested a full tablet report, sending...
18:22:13.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:13.538738 19524 ts_manager.cc:194] Re-registered known tserver with Master: 708d740e01414324ad249bf1854e9e5e (127.18.238.129:41173)
18:22:13.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:13.539588 19624 ts_manager.cc:194] Re-registered known tserver with Master: 708d740e01414324ad249bf1854e9e5e (127.18.238.129:41173)
18:22:13.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:13.541744 19440 ts_manager.cc:194] Re-registered known tserver with Master: 708d740e01414324ad249bf1854e9e5e (127.18.238.129:41173)
18:22:13.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:13.543551 19624 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.18.238.129:51965
18:22:13.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:13.546823 21175 heartbeater.cc:499] Master 127.18.238.188:40453 was elected leader, sending a full tablet report...
18:22:13.566 [DEBUG - kudu-nio-6] (Connection.java:239) [peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)] handling channelInactive
18:22:13.566 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)] cleaning up while in state READY due to: connection closed
18:22:13.567 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
18:22:13.568 [DEBUG - Time-limited test] (Connection.java:748) [peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)] connecting to peer
18:22:13.569 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737
18:22:13.569 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)] cleaning up while in state CONNECTING due to: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737
18:22:13.570 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737) for tablet a6f8d542eda4472e9df5c193be9aa15d: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737
18:22:13.578 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:22:13.579 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)[L],489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557),708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)]}]
18:22:13.579 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=a6f8d542eda4472e9df5c193be9aa15d, attempt=2, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=11), Traces: [0ms] sending RPC to server 24001d6c96764c04999eb5de15b9eba9, [1ms] received response from server 24001d6c96764c04999eb5de15b9eba9: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [2ms] delaying RPC due to: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [10ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=Deferred@571518461(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007f5c7c36b7a8@2c875945 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7b83b558 -> wakeup thread Time-limited test)) after lookup
18:22:13.580 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)] connecting to peer
18:22:13.581 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737
18:22:13.581 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)] cleaning up while in state CONNECTING due to: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737
18:22:13.581 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737) for tablet a6f8d542eda4472e9df5c193be9aa15d: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737
18:22:13.597 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:22:13.597 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)[L],489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557),708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)]}]
18:22:13.598 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=a6f8d542eda4472e9df5c193be9aa15d, attempt=4, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=30), Traces: [0ms] sending RPC to server 24001d6c96764c04999eb5de15b9eba9, [1ms] received response from server 24001d6c96764c04999eb5de15b9eba9: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [2ms] delaying RPC due to: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [10ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [11ms] sending RPC to server 24001d6c96764c04999eb5de15b9eba9, [13ms] received response from server 24001d6c96764c04999eb5de15b9eba9: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [13ms] delaying RPC due to: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [29ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=Deferred@571518461(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007f5c7c36b7a8@2c875945 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@36654418 after retry RPC@1692406074), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7b83b558 -> wakeup thread Time-limited test -> (continuation of Deferred@36654418 after retry RPC@1692406074))) after lookup
18:22:13.598 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)] connecting to peer
18:22:13.599 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737
18:22:13.599 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)] cleaning up while in state CONNECTING due to: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737
18:22:13.599 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737) for tablet a6f8d542eda4472e9df5c193be9aa15d: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737
18:22:13.638 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:22:13.639 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)[L],489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557),708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)]}]
18:22:13.639 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=a6f8d542eda4472e9df5c193be9aa15d, attempt=6, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=71), Traces: [0ms] sending RPC to server 24001d6c96764c04999eb5de15b9eba9, [1ms] received response from server 24001d6c96764c04999eb5de15b9eba9: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [2ms] delaying RPC due to: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [10ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [11ms] sending RPC to server 24001d6c96764c04999eb5de15b9eba9, [13ms] received response from server 24001d6c96764c04999eb5de15b9eba9: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [13ms] delaying RPC due to: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [29ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [30ms] sending RPC to server 24001d6c96764c04999eb5de15b9eba9, [31ms] received response from server 24001d6c96764c04999eb5de15b9eba9: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [31ms] delaying RPC due to: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [66ms] refreshing cache from master, [67ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [70ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=Deferred@571518461(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007f5c7c36b7a8@2c875945 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@36654418 after retry RPC@1692406074) -> (continuation of Deferred@2055596491 after retry RPC@503860832), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7b83b558 -> wakeup thread Time-limited test -> (continuation of Deferred@36654418 after retry RPC@1692406074) -> (continuation of Deferred@2055596491 after retry RPC@503860832))) after lookup
18:22:13.639 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)] connecting to peer
18:22:13.640 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737
18:22:13.640 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)] cleaning up while in state CONNECTING due to: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737
18:22:13.641 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737) for tablet a6f8d542eda4472e9df5c193be9aa15d: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737
18:22:13.698 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:22:13.699 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)[L],489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557),708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)]}]
18:22:13.699 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=a6f8d542eda4472e9df5c193be9aa15d, attempt=8, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=131), Traces: [0ms] sending RPC to server 24001d6c96764c04999eb5de15b9eba9, [1ms] received response from server 24001d6c96764c04999eb5de15b9eba9: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [2ms] delaying RPC due to: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [10ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [11ms] sending RPC to server 24001d6c96764c04999eb5de15b9eba9, [13ms] received response from server 24001d6c96764c04999eb5de15b9eba9: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [13ms] delaying RPC due to: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [29ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [30ms] sending RPC to server 24001d6c96764c04999eb5de15b9eba9, [31ms] received response from server 24001d6c96764c04999eb5de15b9eba9: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [31ms] delaying RPC due to: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [66ms] refreshing cache from master, [67ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [70ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [71ms] sending RPC to server 24001d6c96764c04999eb5de15b9eba9, [72ms] received response from server 24001d6c96764c04999eb5de15b9eba9: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [73ms] delaying RPC due to: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [126ms] refreshing cache from master, [127ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [130ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=Deferred@571518461(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007f5c7c36b7a8@2c875945 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@36654418 after retry RPC@1692406074) -> (continuation of Deferred@2055596491 after retry RPC@503860832) -> (continuation of Deferred@1913826313 after retry RPC@154172974), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7b83b558 -> wakeup thread Time-limited test -> (continuation of Deferred@36654418 after retry RPC@1692406074) -> (continuation of Deferred@2055596491 after retry RPC@503860832) -> (continuation of Deferred@1913826313 after retry RPC@154172974))) after lookup
18:22:13.700 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)] connecting to peer
18:22:13.701 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737
18:22:13.701 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)] cleaning up while in state CONNECTING due to: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737
18:22:13.701 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737) for tablet a6f8d542eda4472e9df5c193be9aa15d: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737
18:22:14.079 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:22:14.079 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)[L],489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557),708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)]}]
18:22:14.080 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=a6f8d542eda4472e9df5c193be9aa15d, attempt=10, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=512), Traces: [0ms] sending RPC to server 24001d6c96764c04999eb5de15b9eba9, [1ms] received response from server 24001d6c96764c04999eb5de15b9eba9: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [2ms] delaying RPC due to: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [10ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [11ms] sending RPC to server 24001d6c96764c04999eb5de15b9eba9, [13ms] received response from server 24001d6c96764c04999eb5de15b9eba9: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [13ms] delaying RPC due to: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [29ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [30ms] sending RPC to server 24001d6c96764c04999eb5de15b9eba9, [31ms] received response from server 24001d6c96764c04999eb5de15b9eba9: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [31ms] delaying RPC due to: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [66ms] refreshing cache from master, [67ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [70ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [71ms] sending RPC to server 24001d6c96764c04999eb5de15b9eba9, [72ms] received response from server 24001d6c96764c04999eb5de15b9eba9: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [73ms] delaying RPC due to: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [126ms] refreshing cache from master, [127ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [130ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [131ms] sending RPC to server 24001d6c96764c04999eb5de15b9eba9, [133ms] received response from server 24001d6c96764c04999eb5de15b9eba9: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [133ms] delaying RPC due to: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [506ms] refreshing cache from master, [507ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [510ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=Deferred@571518461(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007f5c7c36b7a8@2c875945 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@36654418 after retry RPC@1692406074) -> (continuation of Deferred@2055596491 after retry RPC@503860832) -> (continuation of Deferred@1913826313 after retry RPC@154172974) -> (continuation of Deferred@198493222 after retry RPC@27200124), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7b83b558 -> wakeup thread Time-limited test -> (continuation of Deferred@36654418 after retry RPC@1692406074) -> (continuation of Deferred@2055596491 after retry RPC@503860832) -> (continuation of Deferred@1913826313 after retry RPC@154172974) -> (continuation of Deferred@198493222 after retry RPC@27200124))) after lookup
18:22:14.081 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)] connecting to peer
18:22:14.082 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737
18:22:14.082 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)] cleaning up while in state CONNECTING due to: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737
18:22:14.082 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737) for tablet a6f8d542eda4472e9df5c193be9aa15d: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737
18:22:14.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:14.769042 21188 raft_consensus.cc:493] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
18:22:14.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:14.769470 21188 raft_consensus.cc:515] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:22:14.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:14.771802 21188 leader_election.cc:290] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 24001d6c96764c04999eb5de15b9eba9 (127.18.238.130:35737), 489d187f67e24039b9c5ea35f7d1e11e (127.18.238.131:46557)
18:22:14.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:14.776320 21047 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.238.130:35737: connect: Connection refused (error 111)
18:22:14.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:14.781045 21047 leader_election.cc:336] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 24001d6c96764c04999eb5de15b9eba9 (127.18.238.130:35737): Network error: Client connection negotiation failed: client connection to 127.18.238.130:35737: connect: Connection refused (error 111)
18:22:14.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:14.784534 20911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6f8d542eda4472e9df5c193be9aa15d" candidate_uuid: "708d740e01414324ad249bf1854e9e5e" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "489d187f67e24039b9c5ea35f7d1e11e" is_pre_election: true
18:22:14.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:14.786090 21045 leader_election.cc:304] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 708d740e01414324ad249bf1854e9e5e; no voters: 24001d6c96764c04999eb5de15b9eba9, 489d187f67e24039b9c5ea35f7d1e11e
18:22:14.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:14.786782 21188 raft_consensus.cc:2749] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
18:22:14.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:14.910970 21194 raft_consensus.cc:493] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 24001d6c96764c04999eb5de15b9eba9)
18:22:14.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:14.911384 21194 raft_consensus.cc:515] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:22:14.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:14.913039 21194 leader_election.cc:290] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 24001d6c96764c04999eb5de15b9eba9 (127.18.238.130:35737), 708d740e01414324ad249bf1854e9e5e (127.18.238.129:41173)
18:22:14.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:14.919095 20833 leader_election.cc:336] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 24001d6c96764c04999eb5de15b9eba9 (127.18.238.130:35737): Network error: Client connection negotiation failed: client connection to 127.18.238.130:35737: connect: Connection refused (error 111)
18:22:14.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:14.925565 21124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6f8d542eda4472e9df5c193be9aa15d" candidate_uuid: "489d187f67e24039b9c5ea35f7d1e11e" candidate_term: 6 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "708d740e01414324ad249bf1854e9e5e" is_pre_election: true
18:22:14.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:14.926374 21124 raft_consensus.cc:2468] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 489d187f67e24039b9c5ea35f7d1e11e in term 4.
18:22:14.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:14.927817 20833 leader_election.cc:304] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 489d187f67e24039b9c5ea35f7d1e11e, 708d740e01414324ad249bf1854e9e5e; no voters: 24001d6c96764c04999eb5de15b9eba9
18:22:14.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:14.928570 21194 raft_consensus.cc:2804] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 5 FOLLOWER]: Leader pre-election won for term 6
18:22:14.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:14.928886 21194 raft_consensus.cc:493] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 24001d6c96764c04999eb5de15b9eba9)
18:22:14.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:14.929239 21194 raft_consensus.cc:3060] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 5 FOLLOWER]: Advancing to term 6
18:22:14.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:14.933917 21194 raft_consensus.cc:515] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:22:14.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:14.935864 21194 leader_election.cc:290] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [CANDIDATE]: Term 6 election: Requested vote from peers 24001d6c96764c04999eb5de15b9eba9 (127.18.238.130:35737), 708d740e01414324ad249bf1854e9e5e (127.18.238.129:41173)
18:22:14.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:14.938242 21124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6f8d542eda4472e9df5c193be9aa15d" candidate_uuid: "489d187f67e24039b9c5ea35f7d1e11e" candidate_term: 6 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "708d740e01414324ad249bf1854e9e5e"
18:22:14.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:14.938687 21124 raft_consensus.cc:3060] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [term 4 FOLLOWER]: Advancing to term 6
18:22:14.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:14.939944 20833 leader_election.cc:336] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 24001d6c96764c04999eb5de15b9eba9 (127.18.238.130:35737): Network error: Client connection negotiation failed: client connection to 127.18.238.130:35737: connect: Connection refused (error 111)
18:22:14.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:14.945220 21124 raft_consensus.cc:2468] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 489d187f67e24039b9c5ea35f7d1e11e in term 6.
18:22:14.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:14.946128 20833 leader_election.cc:304] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 489d187f67e24039b9c5ea35f7d1e11e, 708d740e01414324ad249bf1854e9e5e; no voters: 24001d6c96764c04999eb5de15b9eba9
18:22:14.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:14.946767 21194 raft_consensus.cc:2804] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 6 FOLLOWER]: Leader election won for term 6
18:22:14.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:14.949024 21194 raft_consensus.cc:697] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 6 LEADER]: Becoming Leader. State: Replica: 489d187f67e24039b9c5ea35f7d1e11e, State: Running, Role: LEADER
18:22:14.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:14.950191 21194 consensus_queue.cc:237] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 5.20, Last appended by leader: 20, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:22:14.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:14.958995 19624 catalog_manager.cc:5719] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e reported cstate change: term changed from 5 to 6, leader changed from 24001d6c96764c04999eb5de15b9eba9 (127.18.238.130) to 489d187f67e24039b9c5ea35f7d1e11e (127.18.238.131). New cstate: current_term: 6 leader_uuid: "489d187f67e24039b9c5ea35f7d1e11e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } health_report { overall_health: UNKNOWN } } }
18:22:15.338 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:22:15.339 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737),489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)[L],708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)]}]
18:22:15.339 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=a6f8d542eda4472e9df5c193be9aa15d, attempt=12, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=1771), Traces: [0ms] sending RPC to server 24001d6c96764c04999eb5de15b9eba9, [1ms] received response from server 24001d6c96764c04999eb5de15b9eba9: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [2ms] delaying RPC due to: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [10ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [11ms] sending RPC to server 24001d6c96764c04999eb5de15b9eba9, [13ms] received response from server 24001d6c96764c04999eb5de15b9eba9: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [13ms] delaying RPC due to: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [29ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [30ms] sending RPC to server 24001d6c96764c04999eb5de15b9eba9, [31ms] received response from server 24001d6c96764c04999eb5de15b9eba9: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [31ms] delaying RPC due to: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [66ms] refreshing cache from master, [67ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [70ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [71ms] sending RPC to server 24001d6c96764c04999eb5de15b9eba9, [72ms] received response from server 24001d6c96764c04999eb5de15b9eba9: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [73ms] delaying RPC due to: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [126ms] refreshing cache from master, [127ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [130ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [131ms] sending RPC to server 24001d6c96764c04999eb5de15b9eba9, [133ms] received response from server 24001d6c96764c04999eb5de15b9eba9: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [133ms] delaying RPC due to: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [506ms] refreshing cache from master, [507ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [510ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [512ms] sending RPC to server 24001d6c96764c04999eb5de15b9eba9, [514ms] received response from server 24001d6c96764c04999eb5de15b9eba9: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [514ms] delaying RPC due to: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [1766ms] refreshing cache from master, [1766ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [1770ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=Deferred@571518461(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007f5c7c36b7a8@2c875945 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@36654418 after retry RPC@1692406074) -> (continuation of Deferred@2055596491 after retry RPC@503860832) -> (continuation of Deferred@1913826313 after retry RPC@154172974) -> (continuation of Deferred@198493222 after retry RPC@27200124) -> (continuation of Deferred@694058586 after retry RPC@1357598183), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7b83b558 -> wakeup thread Time-limited test -> (continuation of Deferred@36654418 after retry RPC@1692406074) -> (continuation of Deferred@2055596491 after retry RPC@503860832) -> (continuation of Deferred@1913826313 after retry RPC@154172974) -> (continuation of Deferred@198493222 after retry RPC@27200124) -> (continuation of Deferred@694058586 after retry RPC@1357598183))) after lookup
18:22:15.340 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)] connecting to peer
18:22:15.341 [DEBUG - kudu-nio-5] (Connection.java:757) [peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)] Successfully connected to peer
18:22:15.367 [DEBUG - kudu-nio-5] (Negotiator.java:918) Authenticated connection [id: 0xc4796aeb, L:/127.0.0.1:40258 - R:/127.18.238.131:46557] using TOKEN/null
18:22:15.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:15.378197 21124 raft_consensus.cc:1275] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [term 6 FOLLOWER]: Refusing update from remote peer 489d187f67e24039b9c5ea35f7d1e11e: Log matching property violated. Preceding OpId in replica: term: 4 index: 16. Preceding OpId from leader: term: 6 index: 22. (index mismatch)
18:22:15.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:15.379878 21199 consensus_queue.cc:1048] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [LEADER]: Connected to new peer: Peer: permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 16, Time since last communication: 0.001s
18:22:15.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:15.381539 20833 consensus_peers.cc:597] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e -> Peer 24001d6c96764c04999eb5de15b9eba9 (127.18.238.130:35737): Couldn't send request to peer 24001d6c96764c04999eb5de15b9eba9. Status: Network error: Client connection negotiation failed: client connection to 127.18.238.130:35737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
18:22:15.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:15.396914 21218 mvcc.cc:204] Tried to move back new op lower bound from 7317391488657068032 to 7317391487967379456. Current Snapshot: MvccSnapshot[applied={T|T < 7317391488657068032}]
18:22:15.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:15.405750 21215 mvcc.cc:204] Tried to move back new op lower bound from 7317391504892723200 to 7317391503164739584. Current Snapshot: MvccSnapshot[applied={T|T < 7317391504892723200}]
18:22:15.459 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.18.238.130:35737
18:22:15.461 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

18:22:15.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:15.462684 19386 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu
18:22:15.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu
18:22:15.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-1/wal
18:22:15.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-1/data
18:22:15.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
18:22:15.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
18:22:15.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
18:22:15.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:22:15.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:22:15.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:22:15.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-1/logs
18:22:15.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-1/data/info.pb
18:22:15.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:22:15.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
18:22:15.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
18:22:15.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
18:22:15.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
18:22:15.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:22:15.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:22:15.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:22:15.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
18:22:15.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
18:22:15.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.18.238.130:35737
18:22:15.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.18.238.130
18:22:15.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.18.238.190:40173,127.18.238.189:35025,127.18.238.188:40453
18:22:15.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44475
18:22:15.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.18.238.130
18:22:15.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.18.238.148:35877
18:22:15.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:22:15.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:22:15.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
18:22:15.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:15.794881 21224 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
18:22:15.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:15.795378 21224 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
18:22:15.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:15.795830 21224 flags.cc:432] Enabled unsafe flag: --never_fsync=true
18:22:15.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:15.831952 21224 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
18:22:15.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:15.833437 21224 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.238.130
18:22:15.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:15.882941 21224 tablet_server_runner.cc:78] Tablet server non-default flags:
18:22:15.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-1/logs
18:22:15.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:22:15.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
18:22:15.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:22:15.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.18.238.148:35877
18:22:15.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:22:15.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
18:22:15.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-1/data
18:22:15.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-1/wal
18:22:15.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:22:15.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:22:15.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.18.238.130:35737
18:22:15.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
18:22:15.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:22:15.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:22:15.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-1/data/info.pb
18:22:15.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.18.238.130
18:22:15.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44475
18:22:15.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.18.238.190:40173,127.18.238.189:35025,127.18.238.188:40453
18:22:15.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
18:22:15.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:22:15.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
18:22:15.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
18:22:15.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:22:15.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.18.238.130
18:22:15.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
18:22:15.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
18:22:15.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
18:22:15.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision cfdbd367ce7fd9747cb6f1860fc64263d9f9f79a
18:22:15.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
18:22:15.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 11 Aug 2026 18:01:34 UTC on bdcb31816ec0
18:22:15.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 17632
18:22:15.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
18:22:15.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:15.884866 21224 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
18:22:15.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:15.887094 21224 file_cache.cc:504] Constructed file cache file cache with capacity 419430
18:22:15.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:15.901973 21231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:22:16.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:16.379256 20833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.238.130:35737: connect: Connection refused (error 111) [suppressed 4 similar messages]
18:22:17.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:17.305505 21230 debug-util.cc:398] Leaking SignalData structure 0x7b080002ede0 after lost signal to thread 21224
18:22:17.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:17.497613 21230 kernel_stack_watchdog.cc:198] Thread 21224 stuck at /home/jenkins-slave/workspace/build_and_test@4/src/kudu/util/thread.cc:642 for 399ms:
18:22:17.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Kernel stack:
18:22:17.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) (could not read kernel stack)
18:22:17.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) User stack:
18:22:17.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) <Timed out: thread did not respond: maybe it is blocking signals>
18:22:17.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:15.903262 21232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:22:17.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:17.498157 21224 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.595s	user 0.512s	sys 1.037s
18:22:17.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:17.498494 21224 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.595s	user 0.512s	sys 1.037s
18:22:17.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:17.500846 21234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:22:17.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:17.503116 21233 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1599 milliseconds
18:22:17.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:17.503120 21224 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
18:22:17.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:17.504446 21224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
18:22:17.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:17.506471 21224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
18:22:17.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:17.507829 21224 hybrid_clock.cc:648] HybridClock initialized: now 1786472537507760 us; error 64 us; skew 500 ppm
18:22:17.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:17.508647 21224 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
18:22:17.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:17.514773 21224 webserver.cc:533] Webserver started at http://127.18.238.130:44475/ using document root <none> and password file <none>
18:22:17.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:17.515691 21224 fs_manager.cc:362] Metadata directory not provided
18:22:17.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:17.515944 21224 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
18:22:17.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:17.523901 21224 fs_manager.cc:714] Time spent opening directory manager: real 0.005s	user 0.000s	sys 0.004s
18:22:17.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:17.528316 21249 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
18:22:17.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:17.529244 21224 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
18:22:17.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:17.529575 21224 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-1/data,/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-1/wal
18:22:17.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "24001d6c96764c04999eb5de15b9eba9"
18:22:17.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 18:21:50 on dist-test-slave-6nmv"
18:22:17.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:17.531579 21224 fs_report.cc:389] FS layout report
18:22:17.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
18:22:17.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-1/wal
18:22:17.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-1/wal
18:22:17.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-1/data/data
18:22:17.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
18:22:17.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
18:22:17.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
18:22:17.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
18:22:17.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
18:22:17.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
18:22:17.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
18:22:17.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
18:22:17.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
18:22:17.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
18:22:17.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
18:22:17.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:17.591080 21224 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
18:22:17.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:17.592623 21224 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
18:22:17.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:17.593187 21224 kserver.cc:163] Server-wide thread pool size limit: 3276
18:22:17.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:17.595988 21224 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
18:22:17.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:17.601603 21258 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
18:22:17.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:17.609959 21224 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
18:22:17.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:17.610209 21224 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.010s	user 0.001s	sys 0.000s
18:22:17.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:17.610574 21224 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
18:22:17.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:17.615231 21224 ts_tablet_manager.cc:616] Registered 1 tablets
18:22:17.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:17.615432 21224 ts_tablet_manager.cc:595] Time spent register tablets: real 0.005s	user 0.004s	sys 0.000s
18:22:17.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:17.615707 21258 tablet_bootstrap.cc:492] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9: Bootstrap starting.
18:22:17.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:17.677680 21258 log.cc:826] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9: Log is configured to *not* fsync() on all Append() calls
18:22:17.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:17.748019 20833 consensus_peers.cc:597] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e -> Peer 24001d6c96764c04999eb5de15b9eba9 (127.18.238.130:35737): Couldn't send request to peer 24001d6c96764c04999eb5de15b9eba9. Status: Network error: Client connection negotiation failed: client connection to 127.18.238.130:35737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
18:22:17.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:17.768431 21258 tablet_bootstrap.cc:492] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9: Bootstrap replayed 1/1 log segments. Stats: ops{read=20 overwritten=0 applied=20 ignored=0} inserts{seen=15 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
18:22:17.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:17.769425 21258 tablet_bootstrap.cc:492] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9: Bootstrap complete.
18:22:17.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:17.771003 21258 ts_tablet_manager.cc:1403] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9: Time spent bootstrapping tablet: real 0.156s	user 0.111s	sys 0.037s
18:22:17.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:17.792218 21258 raft_consensus.cc:359] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:22:17.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:17.795872 21258 raft_consensus.cc:740] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24001d6c96764c04999eb5de15b9eba9, State: Initialized, Role: FOLLOWER
18:22:17.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:17.796962 21258 consensus_queue.cc:260] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 20, Last appended: 5.20, Last appended by leader: 20, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:22:17.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:17.801709 21258 ts_tablet_manager.cc:1434] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9: Time spent starting tablet: real 0.030s	user 0.025s	sys 0.004s
18:22:17.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:17.818334 21224 rpc_server.cc:307] RPC server started. Bound to: 127.18.238.130:35737
18:22:17.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:17.818636 21381 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.238.130:35737 every 8 connection(s)
18:22:17.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:17.821892 21224 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-1/data/info.pb
18:22:17.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:17.827355 19386 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu as pid 21224
18:22:17.828 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
18:22:17.871 [DEBUG - kudu-nio-5] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 18 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897, tablet=a6f8d542eda4472e9df5c193be9aa15d, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
18:22:17.872 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.18.238.131:46557
18:22:17.872 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

18:22:17.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:17.880857 19386 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu with pid 20805
18:22:17.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:17.894766 21385 heartbeater.cc:344] Connected to a master server at 127.18.238.190:40173
18:22:17.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:17.895259 21385 heartbeater.cc:461] Registering TS with master...
18:22:17.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:17.896306 21384 heartbeater.cc:344] Connected to a master server at 127.18.238.188:40453
18:22:17.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:17.896476 21385 heartbeater.cc:507] Master 127.18.238.190:40173 requested a full tablet report, sending...
18:22:17.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:17.896638 21384 heartbeater.cc:461] Registering TS with master...
18:22:17.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:17.897547 21384 heartbeater.cc:507] Master 127.18.238.188:40453 requested a full tablet report, sending...
18:22:17.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:17.904428 19624 ts_manager.cc:194] Re-registered known tserver with Master: 24001d6c96764c04999eb5de15b9eba9 (127.18.238.130:35737)
18:22:17.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:17.904385 19441 ts_manager.cc:194] Re-registered known tserver with Master: 24001d6c96764c04999eb5de15b9eba9 (127.18.238.130:35737)
18:22:17.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:17.906970 19624 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.18.238.130:50353
18:22:17.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:17.914300 21383 heartbeater.cc:344] Connected to a master server at 127.18.238.189:35025
18:22:17.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:17.914654 21383 heartbeater.cc:461] Registering TS with master...
18:22:17.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:17.915315 21383 heartbeater.cc:507] Master 127.18.238.189:35025 requested a full tablet report, sending...
18:22:17.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:17.918881 19524 ts_manager.cc:194] Re-registered known tserver with Master: 24001d6c96764c04999eb5de15b9eba9 (127.18.238.130:35737)
18:22:17.943 [DEBUG - kudu-nio-5] (Connection.java:239) [peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)] handling channelInactive
18:22:17.943 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)] cleaning up while in state READY due to: connection closed
18:22:17.944 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
18:22:17.945 [DEBUG - Time-limited test] (Connection.java:748) [peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)] connecting to peer
18:22:17.946 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557
18:22:17.946 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)] cleaning up while in state CONNECTING due to: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557
18:22:17.946 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557) for tablet a6f8d542eda4472e9df5c193be9aa15d: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557
18:22:17.958 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:22:17.958 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737),489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)[L],708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)]}]
18:22:17.959 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=a6f8d542eda4472e9df5c193be9aa15d, attempt=2, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=14), Traces: [0ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [1ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [1ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [13ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=Deferred@122619056(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007f5c7c36b7a8@d41e5cb -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@61510978 -> wakeup thread Time-limited test)) after lookup
18:22:17.959 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)] connecting to peer
18:22:17.960 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557
18:22:17.960 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)] cleaning up while in state CONNECTING due to: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557
18:22:17.960 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557) for tablet a6f8d542eda4472e9df5c193be9aa15d: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557
18:22:17.977 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:22:17.977 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737),489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)[L],708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)]}]
18:22:17.977 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=a6f8d542eda4472e9df5c193be9aa15d, attempt=4, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=32), Traces: [0ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [1ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [1ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [13ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [14ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [15ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [16ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [32ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=Deferred@122619056(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007f5c7c36b7a8@d41e5cb -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1849818732 after retry RPC@1196012875), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@61510978 -> wakeup thread Time-limited test -> (continuation of Deferred@1849818732 after retry RPC@1196012875))) after lookup
18:22:17.978 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)] connecting to peer
18:22:17.979 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557
18:22:17.979 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)] cleaning up while in state CONNECTING due to: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557
18:22:17.979 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557) for tablet a6f8d542eda4472e9df5c193be9aa15d: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557
18:22:17.997 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:22:17.998 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737),489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)[L],708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)]}]
18:22:17.998 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=a6f8d542eda4472e9df5c193be9aa15d, attempt=6, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=53), Traces: [0ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [1ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [1ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [13ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [14ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [15ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [16ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [32ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [32ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [34ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [34ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [52ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=Deferred@122619056(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007f5c7c36b7a8@d41e5cb -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1849818732 after retry RPC@1196012875) -> (continuation of Deferred@403593661 after retry RPC@317150234), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@61510978 -> wakeup thread Time-limited test -> (continuation of Deferred@1849818732 after retry RPC@1196012875) -> (continuation of Deferred@403593661 after retry RPC@317150234))) after lookup
18:22:17.999 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)] connecting to peer
18:22:17.999 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557
18:22:18.000 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)] cleaning up while in state CONNECTING due to: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557
18:22:18.000 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557) for tablet a6f8d542eda4472e9df5c193be9aa15d: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557
18:22:18.059 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:22:18.059 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737),489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)[L],708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)]}]
18:22:18.060 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=a6f8d542eda4472e9df5c193be9aa15d, attempt=8, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=115), Traces: [0ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [1ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [1ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [13ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [14ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [15ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [16ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [32ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [32ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [34ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [34ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [52ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [53ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [55ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [55ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [109ms] refreshing cache from master, [109ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [113ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=Deferred@122619056(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007f5c7c36b7a8@d41e5cb -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1849818732 after retry RPC@1196012875) -> (continuation of Deferred@403593661 after retry RPC@317150234) -> (continuation of Deferred@1508717495 after retry RPC@1904409364), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@61510978 -> wakeup thread Time-limited test -> (continuation of Deferred@1849818732 after retry RPC@1196012875) -> (continuation of Deferred@403593661 after retry RPC@317150234) -> (continuation of Deferred@1508717495 after retry RPC@1904409364))) after lookup
18:22:18.060 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)] connecting to peer
18:22:18.062 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557
18:22:18.062 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)] cleaning up while in state CONNECTING due to: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557
18:22:18.062 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557) for tablet a6f8d542eda4472e9df5c193be9aa15d: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557
18:22:18.158 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:22:18.159 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737),489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)[L],708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)]}]
18:22:18.159 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=a6f8d542eda4472e9df5c193be9aa15d, attempt=10, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=214), Traces: [0ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [1ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [1ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [13ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [14ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [15ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [16ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [32ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [32ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [34ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [34ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [52ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [53ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [55ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [55ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [109ms] refreshing cache from master, [109ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [113ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [115ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [117ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [117ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [209ms] refreshing cache from master, [209ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [213ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=Deferred@122619056(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007f5c7c36b7a8@d41e5cb -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1849818732 after retry RPC@1196012875) -> (continuation of Deferred@403593661 after retry RPC@317150234) -> (continuation of Deferred@1508717495 after retry RPC@1904409364) -> (continuation of Deferred@1854693017 after retry RPC@462257370), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@61510978 -> wakeup thread Time-limited test -> (continuation of Deferred@1849818732 after retry RPC@1196012875) -> (continuation of Deferred@403593661 after retry RPC@317150234) -> (continuation of Deferred@1508717495 after retry RPC@1904409364) -> (continuation of Deferred@1854693017 after retry RPC@462257370))) after lookup
18:22:18.160 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)] connecting to peer
18:22:18.161 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557
18:22:18.161 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)] cleaning up while in state CONNECTING due to: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557
18:22:18.161 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557) for tablet a6f8d542eda4472e9df5c193be9aa15d: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557
18:22:18.299 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:22:18.299 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737),489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)[L],708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)]}]
18:22:18.299 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=a6f8d542eda4472e9df5c193be9aa15d, attempt=12, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=354), Traces: [0ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [1ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [1ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [13ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [14ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [15ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [16ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [32ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [32ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [34ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [34ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [52ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [53ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [55ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [55ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [109ms] refreshing cache from master, [109ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [113ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [115ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [117ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [117ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [209ms] refreshing cache from master, [209ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [213ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [214ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [216ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [216ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [350ms] refreshing cache from master, [350ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [353ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=Deferred@122619056(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007f5c7c36b7a8@d41e5cb -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1849818732 after retry RPC@1196012875) -> (continuation of Deferred@403593661 after retry RPC@317150234) -> (continuation of Deferred@1508717495 after retry RPC@1904409364) -> (continuation of Deferred@1854693017 after retry RPC@462257370) -> (continuation of Deferred@301484300 after retry RPC@1633913925), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@61510978 -> wakeup thread Time-limited test -> (continuation of Deferred@1849818732 after retry RPC@1196012875) -> (continuation of Deferred@403593661 after retry RPC@317150234) -> (continuation of Deferred@1508717495 after retry RPC@1904409364) -> (continuation of Deferred@1854693017 after retry RPC@462257370) -> (continuation of Deferred@301484300 after retry RPC@1633913925))) after lookup
18:22:18.300 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)] connecting to peer
18:22:18.301 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557
18:22:18.302 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)] cleaning up while in state CONNECTING due to: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557
18:22:18.302 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557) for tablet a6f8d542eda4472e9df5c193be9aa15d: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557
18:22:18.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:18.910306 21384 heartbeater.cc:499] Master 127.18.238.188:40453 was elected leader, sending a full tablet report...
18:22:18.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:18.991573 21399 raft_consensus.cc:493] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
18:22:18.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:18.992202 21399 raft_consensus.cc:515] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:22:18.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:18.995159 21399 leader_election.cc:290] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 489d187f67e24039b9c5ea35f7d1e11e (127.18.238.131:46557), 708d740e01414324ad249bf1854e9e5e (127.18.238.129:41173)
18:22:18.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:18.999320 21251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.238.131:46557: connect: Connection refused (error 111)
18:22:19.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:19.003350 21251 leader_election.cc:336] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 489d187f67e24039b9c5ea35f7d1e11e (127.18.238.131:46557): Network error: Client connection negotiation failed: client connection to 127.18.238.131:46557: connect: Connection refused (error 111)
18:22:19.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:19.007197 21124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6f8d542eda4472e9df5c193be9aa15d" candidate_uuid: "24001d6c96764c04999eb5de15b9eba9" candidate_term: 6 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "708d740e01414324ad249bf1854e9e5e" is_pre_election: true
18:22:19.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:19.008865 21253 leader_election.cc:304] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24001d6c96764c04999eb5de15b9eba9; no voters: 489d187f67e24039b9c5ea35f7d1e11e, 708d740e01414324ad249bf1854e9e5e
18:22:19.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:19.009552 21399 raft_consensus.cc:2749] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
18:22:19.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:19.069965 21404 raft_consensus.cc:493] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 489d187f67e24039b9c5ea35f7d1e11e)
18:22:19.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:19.070467 21404 raft_consensus.cc:515] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:22:19.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:19.072165 21404 leader_election.cc:290] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 24001d6c96764c04999eb5de15b9eba9 (127.18.238.130:35737), 489d187f67e24039b9c5ea35f7d1e11e (127.18.238.131:46557)
18:22:19.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:19.079078 21045 leader_election.cc:336] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 489d187f67e24039b9c5ea35f7d1e11e (127.18.238.131:46557): Network error: Client connection negotiation failed: client connection to 127.18.238.131:46557: connect: Connection refused (error 111)
18:22:19.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:19.084295 21330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6f8d542eda4472e9df5c193be9aa15d" candidate_uuid: "708d740e01414324ad249bf1854e9e5e" candidate_term: 7 candidate_status { last_received { term: 6 index: 24 } } ignore_live_leader: false dest_uuid: "24001d6c96764c04999eb5de15b9eba9" is_pre_election: true
18:22:19.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:19.084944 21330 raft_consensus.cc:2468] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 708d740e01414324ad249bf1854e9e5e in term 5.
18:22:19.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:19.086153 21047 leader_election.cc:304] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24001d6c96764c04999eb5de15b9eba9, 708d740e01414324ad249bf1854e9e5e; no voters: 489d187f67e24039b9c5ea35f7d1e11e
18:22:19.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:19.086781 21404 raft_consensus.cc:2804] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [term 6 FOLLOWER]: Leader pre-election won for term 7
18:22:19.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:19.087018 21404 raft_consensus.cc:493] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [term 6 FOLLOWER]: Starting leader election (detected failure of leader 489d187f67e24039b9c5ea35f7d1e11e)
18:22:19.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:19.087244 21404 raft_consensus.cc:3060] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [term 6 FOLLOWER]: Advancing to term 7
18:22:19.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:19.092252 21404 raft_consensus.cc:515] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:22:19.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:19.093849 21404 leader_election.cc:290] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [CANDIDATE]: Term 7 election: Requested vote from peers 24001d6c96764c04999eb5de15b9eba9 (127.18.238.130:35737), 489d187f67e24039b9c5ea35f7d1e11e (127.18.238.131:46557)
18:22:19.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:19.094580 21330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6f8d542eda4472e9df5c193be9aa15d" candidate_uuid: "708d740e01414324ad249bf1854e9e5e" candidate_term: 7 candidate_status { last_received { term: 6 index: 24 } } ignore_live_leader: false dest_uuid: "24001d6c96764c04999eb5de15b9eba9"
18:22:19.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:19.095139 21330 raft_consensus.cc:3060] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 5 FOLLOWER]: Advancing to term 7
18:22:19.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:19.097875 21045 leader_election.cc:336] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 489d187f67e24039b9c5ea35f7d1e11e (127.18.238.131:46557): Network error: Client connection negotiation failed: client connection to 127.18.238.131:46557: connect: Connection refused (error 111)
18:22:19.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:19.102046 21330 raft_consensus.cc:2468] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 708d740e01414324ad249bf1854e9e5e in term 7.
18:22:19.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:19.102943 21047 leader_election.cc:304] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24001d6c96764c04999eb5de15b9eba9, 708d740e01414324ad249bf1854e9e5e; no voters: 489d187f67e24039b9c5ea35f7d1e11e
18:22:19.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:19.103577 21404 raft_consensus.cc:2804] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [term 7 FOLLOWER]: Leader election won for term 7
18:22:19.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:19.104656 21404 raft_consensus.cc:697] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [term 7 LEADER]: Becoming Leader. State: Replica: 708d740e01414324ad249bf1854e9e5e, State: Running, Role: LEADER
18:22:19.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:19.105489 21404 consensus_queue.cc:237] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 6.24, Last appended by leader: 24, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:22:19.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:19.114332 19624 catalog_manager.cc:5719] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e reported cstate change: term changed from 6 to 7, leader changed from 489d187f67e24039b9c5ea35f7d1e11e (127.18.238.131) to 708d740e01414324ad249bf1854e9e5e (127.18.238.129). New cstate: current_term: 7 leader_uuid: "708d740e01414324ad249bf1854e9e5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } health_report { overall_health: HEALTHY } } }
18:22:19.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:19.519474 21045 consensus_peers.cc:597] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e -> Peer 489d187f67e24039b9c5ea35f7d1e11e (127.18.238.131:46557): Couldn't send request to peer 489d187f67e24039b9c5ea35f7d1e11e. Status: Network error: Client connection negotiation failed: client connection to 127.18.238.131:46557: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
18:22:19.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:19.570014 21330 raft_consensus.cc:1275] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 7 FOLLOWER]: Refusing update from remote peer 708d740e01414324ad249bf1854e9e5e: Log matching property violated. Preceding OpId in replica: term: 5 index: 20. Preceding OpId from leader: term: 7 index: 25. (index mismatch)
18:22:19.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:19.571492 21404 consensus_queue.cc:1048] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 20, Time since last communication: 0.000s
18:22:20.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:20.073709 21045 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.238.131:46557: connect: Connection refused (error 111) [suppressed 3 similar messages]
18:22:22.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:22.025158 21045 consensus_peers.cc:597] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e -> Peer 489d187f67e24039b9c5ea35f7d1e11e (127.18.238.131:46557): Couldn't send request to peer 489d187f67e24039b9c5ea35f7d1e11e. Status: Network error: Client connection negotiation failed: client connection to 127.18.238.131:46557: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
18:22:22.238 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:22:22.238 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737),489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557),708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)[L]]}]
18:22:22.239 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=a6f8d542eda4472e9df5c193be9aa15d, attempt=14, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=4293), Traces: [0ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [1ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [1ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [13ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [14ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [15ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [16ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [32ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [32ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [34ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [34ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [52ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [53ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [55ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [55ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [109ms] refreshing cache from master, [109ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [113ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [115ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [117ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [117ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [209ms] refreshing cache from master, [209ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [213ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [214ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [216ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [216ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [350ms] refreshing cache from master, [350ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [353ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [355ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [357ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [357ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [4289ms] refreshing cache from master, [4289ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [4293ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=Deferred@122619056(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007f5c7c36b7a8@d41e5cb -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1849818732 after retry RPC@1196012875) -> (continuation of Deferred@403593661 after retry RPC@317150234) -> (continuation of Deferred@1508717495 after retry RPC@1904409364) -> (continuation of Deferred@1854693017 after retry RPC@462257370) -> (continuation of Deferred@301484300 after retry RPC@1633913925) -> (continuation of Deferred@183009784 after retry RPC@1997808964), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@61510978 -> wakeup thread Time-limited test -> (continuation of Deferred@1849818732 after retry RPC@1196012875) -> (continuation of Deferred@403593661 after retry RPC@317150234) -> (continuation of Deferred@1508717495 after retry RPC@1904409364) -> (continuation of Deferred@1854693017 after retry RPC@462257370) -> (continuation of Deferred@301484300 after retry RPC@1633913925) -> (continuation of Deferred@183009784 after retry RPC@1997808964))) after lookup
18:22:22.239 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)] connecting to peer
18:22:22.240 [DEBUG - kudu-nio-5] (Connection.java:757) [peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)] Successfully connected to peer
18:22:22.259 [DEBUG - kudu-nio-5] (Negotiator.java:918) Authenticated connection [id: 0xbe4ca287, L:/127.0.0.1:46018 - R:/127.18.238.129:41173] using TOKEN/null
18:22:22.306 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.18.238.131:46557
18:22:22.306 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

18:22:22.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:22.308115 19386 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu
18:22:22.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu
18:22:22.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-2/wal
18:22:22.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-2/data
18:22:22.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
18:22:22.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
18:22:22.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
18:22:22.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:22:22.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:22:22.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:22:22.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-2/logs
18:22:22.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-2/data/info.pb
18:22:22.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:22:22.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
18:22:22.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
18:22:22.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
18:22:22.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
18:22:22.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:22:22.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:22:22.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:22:22.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
18:22:22.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
18:22:22.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.18.238.131:46557
18:22:22.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.18.238.131
18:22:22.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.18.238.190:40173,127.18.238.189:35025,127.18.238.188:40453
18:22:22.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=34759
18:22:22.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.18.238.131
18:22:22.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.18.238.148:35877
18:22:22.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:22:22.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:22:22.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
18:22:22.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:22.633313 21463 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
18:22:22.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:22.633723 21463 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
18:22:22.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:22.634186 21463 flags.cc:432] Enabled unsafe flag: --never_fsync=true
18:22:22.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:22.663549 21463 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
18:22:22.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:22.664290 21463 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.238.131
18:22:22.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:22.703243 21463 tablet_server_runner.cc:78] Tablet server non-default flags:
18:22:22.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-2/logs
18:22:22.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:22:22.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
18:22:22.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:22:22.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.18.238.148:35877
18:22:22.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:22:22.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
18:22:22.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-2/data
18:22:22.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-2/wal
18:22:22.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:22:22.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:22:22.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.18.238.131:46557
18:22:22.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
18:22:22.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:22:22.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:22:22.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-2/data/info.pb
18:22:22.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.18.238.131
18:22:22.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=34759
18:22:22.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.18.238.190:40173,127.18.238.189:35025,127.18.238.188:40453
18:22:22.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
18:22:22.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:22:22.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
18:22:22.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
18:22:22.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:22:22.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.18.238.131
18:22:22.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
18:22:22.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
18:22:22.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
18:22:22.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision cfdbd367ce7fd9747cb6f1860fc64263d9f9f79a
18:22:22.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
18:22:22.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 11 Aug 2026 18:01:34 UTC on bdcb31816ec0
18:22:22.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 17632
18:22:22.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
18:22:22.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:22.704843 21463 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
18:22:22.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:22.706627 21463 file_cache.cc:504] Constructed file cache file cache with capacity 419430
18:22:22.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:22.719569 21472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:22:24.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:24.573335 21045 consensus_peers.cc:597] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e -> Peer 489d187f67e24039b9c5ea35f7d1e11e (127.18.238.131:46557): Couldn't send request to peer 489d187f67e24039b9c5ea35f7d1e11e. Status: Network error: Client connection negotiation failed: client connection to 127.18.238.131:46557: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
18:22:25.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:24.128204 21471 debug-util.cc:398] Leaking SignalData structure 0x7b080002ede0 after lost signal to thread 21463
18:22:25.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:25.349622 21471 kernel_stack_watchdog.cc:198] Thread 21463 stuck at /home/jenkins-slave/workspace/build_and_test@4/src/kudu/util/thread.cc:642 for 400ms:
18:22:25.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Kernel stack:
18:22:25.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) (could not read kernel stack)
18:22:25.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) User stack:
18:22:25.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) <Timed out: thread did not respond: maybe it is blocking signals>
18:22:25.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:22.722357 21474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:22:25.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:25.350332 21463 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 2.630s	user 0.840s	sys 1.513s
18:22:25.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:25.350695 21463 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 2.631s	user 0.840s	sys 1.513s
18:22:25.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:25.352618 21476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:22:25.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:25.354565 21475 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 2629 milliseconds
18:22:25.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:25.354565 21463 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
18:22:25.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:25.355772 21463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
18:22:25.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:25.357609 21463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
18:22:25.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:25.358955 21463 hybrid_clock.cc:648] HybridClock initialized: now 1786472545358921 us; error 30 us; skew 500 ppm
18:22:25.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:25.359709 21463 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
18:22:25.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:25.365522 21463 webserver.cc:533] Webserver started at http://127.18.238.131:34759/ using document root <none> and password file <none>
18:22:25.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:25.366549 21463 fs_manager.cc:362] Metadata directory not provided
18:22:25.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:25.366798 21463 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
18:22:25.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:25.374733 21463 fs_manager.cc:714] Time spent opening directory manager: real 0.005s	user 0.006s	sys 0.001s
18:22:25.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:25.379045 21489 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
18:22:25.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:25.380023 21463 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.003s
18:22:25.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:25.380367 21463 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-2/data,/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-2/wal
18:22:25.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "489d187f67e24039b9c5ea35f7d1e11e"
18:22:25.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 18:21:52 on dist-test-slave-6nmv"
18:22:25.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:25.382723 21463 fs_report.cc:389] FS layout report
18:22:25.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
18:22:25.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-2/wal
18:22:25.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-2/wal
18:22:25.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-2/data/data
18:22:25.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
18:22:25.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
18:22:25.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
18:22:25.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
18:22:25.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
18:22:25.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
18:22:25.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
18:22:25.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
18:22:25.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
18:22:25.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
18:22:25.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
18:22:25.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:25.426766 21463 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
18:22:25.428 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:25.428149 21463 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
18:22:25.428 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:25.428643 21463 kserver.cc:163] Server-wide thread pool size limit: 3276
18:22:25.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:25.431207 21463 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
18:22:25.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:25.436978 21497 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
18:22:25.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:25.445497 21463 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
18:22:25.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:25.445741 21463 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.010s	user 0.001s	sys 0.001s
18:22:25.446 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:25.446126 21463 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
18:22:25.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:25.450827 21463 ts_tablet_manager.cc:616] Registered 1 tablets
18:22:25.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:25.451045 21463 ts_tablet_manager.cc:595] Time spent register tablets: real 0.005s	user 0.004s	sys 0.000s
18:22:25.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:25.451354 21497 tablet_bootstrap.cc:492] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e: Bootstrap starting.
18:22:25.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:25.493526 21045 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.238.131:46557: connect: Connection refused (error 111) [suppressed 10 similar messages]
18:22:25.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:25.528964 21497 log.cc:826] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e: Log is configured to *not* fsync() on all Append() calls
18:22:25.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:25.626364 21463 rpc_server.cc:307] RPC server started. Bound to: 127.18.238.131:46557
18:22:25.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:25.626427 21624 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.238.131:46557 every 8 connection(s)
18:22:25.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:25.629878 21463 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-2/data/info.pb
18:22:25.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:25.634895 21497 tablet_bootstrap.cc:492] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e: Bootstrap replayed 1/1 log segments. Stats: ops{read=24 overwritten=0 applied=24 ignored=0} inserts{seen=18 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
18:22:25.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:25.636075 21497 tablet_bootstrap.cc:492] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e: Bootstrap complete.
18:22:25.639 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
18:22:25.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:25.637405 19386 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu as pid 21463
18:22:25.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:25.638187 21497 ts_tablet_manager.cc:1403] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e: Time spent bootstrapping tablet: real 0.187s	user 0.141s	sys 0.040s
18:22:25.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:25.664873 21497 raft_consensus.cc:359] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 6 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:22:25.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:25.667225 21497 raft_consensus.cc:740] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 489d187f67e24039b9c5ea35f7d1e11e, State: Initialized, Role: FOLLOWER
18:22:25.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:25.667965 21497 consensus_queue.cc:260] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 24, Last appended: 6.24, Last appended by leader: 24, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:22:25.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:25.671311 21497 ts_tablet_manager.cc:1434] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e: Time spent starting tablet: real 0.033s	user 0.028s	sys 0.004s
18:22:25.684 [DEBUG - kudu-nio-5] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 21 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897, tablet=a6f8d542eda4472e9df5c193be9aa15d, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
18:22:25.685 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.18.238.129:41173
18:22:25.685 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

18:22:25.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:25.686256 19386 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu with pid 21017
18:22:25.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:25.711469 21625 heartbeater.cc:344] Connected to a master server at 127.18.238.189:35025
18:22:25.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:25.712033 21625 heartbeater.cc:461] Registering TS with master...
18:22:25.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:25.713379 21625 heartbeater.cc:507] Master 127.18.238.189:35025 requested a full tablet report, sending...
18:22:25.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:25.713711 21627 heartbeater.cc:344] Connected to a master server at 127.18.238.190:40173
18:22:25.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:25.714090 21627 heartbeater.cc:461] Registering TS with master...
18:22:25.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:25.714983 21627 heartbeater.cc:507] Master 127.18.238.190:40173 requested a full tablet report, sending...
18:22:25.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:25.718078 21626 heartbeater.cc:344] Connected to a master server at 127.18.238.188:40453
18:22:25.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:25.718364 21626 heartbeater.cc:461] Registering TS with master...
18:22:25.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:25.718150 19526 ts_manager.cc:194] Re-registered known tserver with Master: 489d187f67e24039b9c5ea35f7d1e11e (127.18.238.131:46557)
18:22:25.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:25.718564 19440 ts_manager.cc:194] Re-registered known tserver with Master: 489d187f67e24039b9c5ea35f7d1e11e (127.18.238.131:46557)
18:22:25.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:25.719303 21626 heartbeater.cc:507] Master 127.18.238.188:40453 requested a full tablet report, sending...
18:22:25.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:25.722540 19625 ts_manager.cc:194] Re-registered known tserver with Master: 489d187f67e24039b9c5ea35f7d1e11e (127.18.238.131:46557)
18:22:25.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:25.725636 19625 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.18.238.131:54631
18:22:25.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:25.730386 21626 heartbeater.cc:499] Master 127.18.238.188:40453 was elected leader, sending a full tablet report...
18:22:25.743 [DEBUG - kudu-nio-5] (Connection.java:239) [peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)] handling channelInactive
18:22:25.743 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)] cleaning up while in state READY due to: connection closed
18:22:25.745 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
18:22:25.746 [DEBUG - Time-limited test] (Connection.java:748) [peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)] connecting to peer
18:22:25.747 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:22:25.747 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)] cleaning up while in state CONNECTING due to: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:22:25.747 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173) for tablet a6f8d542eda4472e9df5c193be9aa15d: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:22:25.758 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:22:25.758 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737),489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557),708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)[L]]}]
18:22:25.759 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=a6f8d542eda4472e9df5c193be9aa15d, attempt=2, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=13), Traces: [0ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [2ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [2ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [13ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=Deferred@1305866353(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007f5c7c36b7a8@1bbdc28a -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5490ba01 -> wakeup thread Time-limited test)) after lookup
18:22:25.759 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)] connecting to peer
18:22:25.760 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:22:25.760 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)] cleaning up while in state CONNECTING due to: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:22:25.761 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173) for tablet a6f8d542eda4472e9df5c193be9aa15d: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:22:25.778 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:22:25.778 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737),489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557),708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)[L]]}]
18:22:25.778 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=a6f8d542eda4472e9df5c193be9aa15d, attempt=4, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=32), Traces: [0ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [2ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [2ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [13ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [14ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [15ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [16ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [32ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=Deferred@1305866353(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007f5c7c36b7a8@1bbdc28a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@575671837 after retry RPC@1933703396), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5490ba01 -> wakeup thread Time-limited test -> (continuation of Deferred@575671837 after retry RPC@1933703396))) after lookup
18:22:25.779 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)] connecting to peer
18:22:25.780 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:22:25.780 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)] cleaning up while in state CONNECTING due to: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:22:25.780 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173) for tablet a6f8d542eda4472e9df5c193be9aa15d: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:22:25.797 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:22:25.798 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737),489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557),708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)[L]]}]
18:22:25.798 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=a6f8d542eda4472e9df5c193be9aa15d, attempt=6, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=52), Traces: [0ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [2ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [2ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [13ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [14ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [15ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [16ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [32ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [33ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [35ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [35ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [52ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=Deferred@1305866353(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007f5c7c36b7a8@1bbdc28a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@575671837 after retry RPC@1933703396) -> (continuation of Deferred@309399497 after retry RPC@1854620603), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5490ba01 -> wakeup thread Time-limited test -> (continuation of Deferred@575671837 after retry RPC@1933703396) -> (continuation of Deferred@309399497 after retry RPC@1854620603))) after lookup
18:22:25.798 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)] connecting to peer
18:22:25.799 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:22:25.799 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)] cleaning up while in state CONNECTING due to: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:22:25.800 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173) for tablet a6f8d542eda4472e9df5c193be9aa15d: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:22:25.859 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:22:25.859 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737),489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557),708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)[L]]}]
18:22:25.859 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=a6f8d542eda4472e9df5c193be9aa15d, attempt=8, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=114), Traces: [0ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [2ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [2ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [13ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [14ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [15ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [16ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [32ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [33ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [35ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [35ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [52ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [53ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [54ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [55ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [109ms] refreshing cache from master, [109ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [113ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=Deferred@1305866353(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007f5c7c36b7a8@1bbdc28a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@575671837 after retry RPC@1933703396) -> (continuation of Deferred@309399497 after retry RPC@1854620603) -> (continuation of Deferred@1772593929 after retry RPC@1569039133), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5490ba01 -> wakeup thread Time-limited test -> (continuation of Deferred@575671837 after retry RPC@1933703396) -> (continuation of Deferred@309399497 after retry RPC@1854620603) -> (continuation of Deferred@1772593929 after retry RPC@1569039133))) after lookup
18:22:25.860 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)] connecting to peer
18:22:25.861 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:22:25.861 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)] cleaning up while in state CONNECTING due to: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:22:25.861 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173) for tablet a6f8d542eda4472e9df5c193be9aa15d: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:22:26.278 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:22:26.278 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737),489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557),708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)[L]]}]
18:22:26.278 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=a6f8d542eda4472e9df5c193be9aa15d, attempt=10, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=533), Traces: [0ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [2ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [2ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [13ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [14ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [15ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [16ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [32ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [33ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [35ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [35ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [52ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [53ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [54ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [55ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [109ms] refreshing cache from master, [109ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [113ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [115ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [116ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [116ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [529ms] refreshing cache from master, [529ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [532ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=Deferred@1305866353(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007f5c7c36b7a8@1bbdc28a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@575671837 after retry RPC@1933703396) -> (continuation of Deferred@309399497 after retry RPC@1854620603) -> (continuation of Deferred@1772593929 after retry RPC@1569039133) -> (continuation of Deferred@1898265393 after retry RPC@1564063958), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5490ba01 -> wakeup thread Time-limited test -> (continuation of Deferred@575671837 after retry RPC@1933703396) -> (continuation of Deferred@309399497 after retry RPC@1854620603) -> (continuation of Deferred@1772593929 after retry RPC@1569039133) -> (continuation of Deferred@1898265393 after retry RPC@1564063958))) after lookup
18:22:26.279 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)] connecting to peer
18:22:26.280 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:22:26.280 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)] cleaning up while in state CONNECTING due to: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:22:26.280 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173) for tablet a6f8d542eda4472e9df5c193be9aa15d: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:22:26.498 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:22:26.498 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737),489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557),708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)[L]]}]
18:22:26.499 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=a6f8d542eda4472e9df5c193be9aa15d, attempt=12, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=753), Traces: [0ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [2ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [2ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [13ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [14ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [15ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [16ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [32ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [33ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [35ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [35ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [52ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [53ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [54ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [55ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [109ms] refreshing cache from master, [109ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [113ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [115ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [116ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [116ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [529ms] refreshing cache from master, [529ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [532ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [534ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [535ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [535ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [749ms] refreshing cache from master, [749ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [753ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=Deferred@1305866353(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007f5c7c36b7a8@1bbdc28a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@575671837 after retry RPC@1933703396) -> (continuation of Deferred@309399497 after retry RPC@1854620603) -> (continuation of Deferred@1772593929 after retry RPC@1569039133) -> (continuation of Deferred@1898265393 after retry RPC@1564063958) -> (continuation of Deferred@467297468 after retry RPC@188059673), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5490ba01 -> wakeup thread Time-limited test -> (continuation of Deferred@575671837 after retry RPC@1933703396) -> (continuation of Deferred@309399497 after retry RPC@1854620603) -> (continuation of Deferred@1772593929 after retry RPC@1569039133) -> (continuation of Deferred@1898265393 after retry RPC@1564063958) -> (continuation of Deferred@467297468 after retry RPC@188059673))) after lookup
18:22:26.499 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)] connecting to peer
18:22:26.500 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:22:26.500 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)] cleaning up while in state CONNECTING due to: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:22:26.500 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173) for tablet a6f8d542eda4472e9df5c193be9aa15d: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:22:27.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:27.000674 21639 raft_consensus.cc:493] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
18:22:27.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:27.001138 21639 raft_consensus.cc:515] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:22:27.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:27.003364 21639 leader_election.cc:290] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 24001d6c96764c04999eb5de15b9eba9 (127.18.238.130:35737), 708d740e01414324ad249bf1854e9e5e (127.18.238.129:41173)
18:22:27.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:27.008615 21492 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.238.129:41173: connect: Connection refused (error 111)
18:22:27.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:27.012907 21492 leader_election.cc:336] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 708d740e01414324ad249bf1854e9e5e (127.18.238.129:41173): Network error: Client connection negotiation failed: client connection to 127.18.238.129:41173: connect: Connection refused (error 111)
18:22:27.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:27.014760 21330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6f8d542eda4472e9df5c193be9aa15d" candidate_uuid: "489d187f67e24039b9c5ea35f7d1e11e" candidate_term: 7 candidate_status { last_received { term: 6 index: 24 } } ignore_live_leader: false dest_uuid: "24001d6c96764c04999eb5de15b9eba9" is_pre_election: true
18:22:27.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:27.016337 21492 leader_election.cc:304] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 489d187f67e24039b9c5ea35f7d1e11e; no voters: 24001d6c96764c04999eb5de15b9eba9, 708d740e01414324ad249bf1854e9e5e
18:22:27.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:27.016958 21639 raft_consensus.cc:2749] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
18:22:27.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:27.099555 21645 raft_consensus.cc:493] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 708d740e01414324ad249bf1854e9e5e)
18:22:27.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:27.100111 21645 raft_consensus.cc:515] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:22:27.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:27.102181 21645 leader_election.cc:290] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 489d187f67e24039b9c5ea35f7d1e11e (127.18.238.131:46557), 708d740e01414324ad249bf1854e9e5e (127.18.238.129:41173)
18:22:27.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:27.105496 21253 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.238.129:41173: connect: Connection refused (error 111)
18:22:27.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:27.109421 21253 leader_election.cc:336] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 708d740e01414324ad249bf1854e9e5e (127.18.238.129:41173): Network error: Client connection negotiation failed: client connection to 127.18.238.129:41173: connect: Connection refused (error 111)
18:22:27.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:27.115064 21576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6f8d542eda4472e9df5c193be9aa15d" candidate_uuid: "24001d6c96764c04999eb5de15b9eba9" candidate_term: 8 candidate_status { last_received { term: 7 index: 28 } } ignore_live_leader: false dest_uuid: "489d187f67e24039b9c5ea35f7d1e11e" is_pre_election: true
18:22:27.116 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:27.115906 21576 raft_consensus.cc:2468] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24001d6c96764c04999eb5de15b9eba9 in term 6.
18:22:27.117 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:27.117417 21251 leader_election.cc:304] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24001d6c96764c04999eb5de15b9eba9, 489d187f67e24039b9c5ea35f7d1e11e; no voters: 708d740e01414324ad249bf1854e9e5e
18:22:27.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:27.118121 21645 raft_consensus.cc:2804] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 7 FOLLOWER]: Leader pre-election won for term 8
18:22:27.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:27.118433 21645 raft_consensus.cc:493] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 708d740e01414324ad249bf1854e9e5e)
18:22:27.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:27.118656 21645 raft_consensus.cc:3060] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 7 FOLLOWER]: Advancing to term 8
18:22:27.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:27.123389 21645 raft_consensus.cc:515] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:22:27.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:27.124907 21645 leader_election.cc:290] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [CANDIDATE]: Term 8 election: Requested vote from peers 489d187f67e24039b9c5ea35f7d1e11e (127.18.238.131:46557), 708d740e01414324ad249bf1854e9e5e (127.18.238.129:41173)
18:22:27.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:27.125692 21576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6f8d542eda4472e9df5c193be9aa15d" candidate_uuid: "24001d6c96764c04999eb5de15b9eba9" candidate_term: 8 candidate_status { last_received { term: 7 index: 28 } } ignore_live_leader: false dest_uuid: "489d187f67e24039b9c5ea35f7d1e11e"
18:22:27.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:27.126263 21576 raft_consensus.cc:3060] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 6 FOLLOWER]: Advancing to term 8
18:22:27.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:27.128870 21253 leader_election.cc:336] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 708d740e01414324ad249bf1854e9e5e (127.18.238.129:41173): Network error: Client connection negotiation failed: client connection to 127.18.238.129:41173: connect: Connection refused (error 111)
18:22:27.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:27.133162 21576 raft_consensus.cc:2468] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24001d6c96764c04999eb5de15b9eba9 in term 8.
18:22:27.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:27.134076 21251 leader_election.cc:304] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24001d6c96764c04999eb5de15b9eba9, 489d187f67e24039b9c5ea35f7d1e11e; no voters: 708d740e01414324ad249bf1854e9e5e
18:22:27.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:27.134752 21645 raft_consensus.cc:2804] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 8 FOLLOWER]: Leader election won for term 8
18:22:27.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:27.136169 21645 raft_consensus.cc:697] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 8 LEADER]: Becoming Leader. State: Replica: 24001d6c96764c04999eb5de15b9eba9, State: Running, Role: LEADER
18:22:27.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:27.137182 21645 consensus_queue.cc:237] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 7.28, Last appended by leader: 28, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:22:27.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:27.147292 19625 catalog_manager.cc:5719] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 reported cstate change: term changed from 7 to 8, leader changed from 708d740e01414324ad249bf1854e9e5e (127.18.238.129) to 24001d6c96764c04999eb5de15b9eba9 (127.18.238.130). New cstate: current_term: 8 leader_uuid: "24001d6c96764c04999eb5de15b9eba9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } health_report { overall_health: UNKNOWN } } }
18:22:27.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:27.519088 21576 raft_consensus.cc:1275] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 8 FOLLOWER]: Refusing update from remote peer 24001d6c96764c04999eb5de15b9eba9: Log matching property violated. Preceding OpId in replica: term: 6 index: 24. Preceding OpId from leader: term: 8 index: 29. (index mismatch)
18:22:27.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:27.520548 21645 consensus_queue.cc:1048] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 24, Time since last communication: 0.000s
18:22:27.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:27.534704 21667 mvcc.cc:204] Tried to move back new op lower bound from 7317391533117374464 to 7317391520184279040. Current Snapshot: MvccSnapshot[applied={T|T < 7317391533117374464}]
18:22:27.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:27.553030 21253 consensus_peers.cc:597] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 -> Peer 708d740e01414324ad249bf1854e9e5e (127.18.238.129:41173): Couldn't send request to peer 708d740e01414324ad249bf1854e9e5e. Status: Network error: Client connection negotiation failed: client connection to 127.18.238.129:41173: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
18:22:27.938 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:22:27.938 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)[L],489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557),708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)]}]
18:22:27.939 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=a6f8d542eda4472e9df5c193be9aa15d, attempt=14, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=2193), Traces: [0ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [2ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [2ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [13ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [14ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [15ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [16ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [32ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [33ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [35ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [35ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [52ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [53ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [54ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [55ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [109ms] refreshing cache from master, [109ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [113ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [115ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [116ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [116ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [529ms] refreshing cache from master, [529ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [532ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [534ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [535ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [535ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [749ms] refreshing cache from master, [749ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [753ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [754ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [755ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [756ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [2189ms] refreshing cache from master, [2189ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [2193ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=Deferred@1305866353(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007f5c7c36b7a8@1bbdc28a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@575671837 after retry RPC@1933703396) -> (continuation of Deferred@309399497 after retry RPC@1854620603) -> (continuation of Deferred@1772593929 after retry RPC@1569039133) -> (continuation of Deferred@1898265393 after retry RPC@1564063958) -> (continuation of Deferred@467297468 after retry RPC@188059673) -> (continuation of Deferred@587693108 after retry RPC@68519189), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5490ba01 -> wakeup thread Time-limited test -> (continuation of Deferred@575671837 after retry RPC@1933703396) -> (continuation of Deferred@309399497 after retry RPC@1854620603) -> (continuation of Deferred@1772593929 after retry RPC@1569039133) -> (continuation of Deferred@1898265393 after retry RPC@1564063958) -> (continuation of Deferred@467297468 after retry RPC@188059673) -> (continuation of Deferred@587693108 after retry RPC@68519189))) after lookup
18:22:27.939 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)] connecting to peer
18:22:27.940 [DEBUG - kudu-nio-5] (Connection.java:757) [peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)] Successfully connected to peer
18:22:27.958 [DEBUG - kudu-nio-5] (Negotiator.java:918) Authenticated connection [id: 0xe22a04e8, L:/127.0.0.1:37796 - R:/127.18.238.130:35737] using TOKEN/null
18:22:28.000 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.18.238.129:41173
18:22:28.000 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

18:22:28.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:28.002002 19386 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu
18:22:28.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu
18:22:28.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/wal
18:22:28.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/data
18:22:28.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
18:22:28.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
18:22:28.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
18:22:28.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:22:28.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:22:28.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:22:28.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/logs
18:22:28.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/data/info.pb
18:22:28.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:22:28.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
18:22:28.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
18:22:28.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
18:22:28.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
18:22:28.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:22:28.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:22:28.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:22:28.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
18:22:28.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
18:22:28.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.18.238.129:41173
18:22:28.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.18.238.129
18:22:28.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.18.238.190:40173,127.18.238.189:35025,127.18.238.188:40453
18:22:28.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44313
18:22:28.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.18.238.129
18:22:28.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.18.238.148:35877
18:22:28.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:22:28.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:22:28.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
18:22:28.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:28.302596 21680 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
18:22:28.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:28.303053 21680 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
18:22:28.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:28.303522 21680 flags.cc:432] Enabled unsafe flag: --never_fsync=true
18:22:28.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:28.334414 21680 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
18:22:28.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:28.335382 21680 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.238.129
18:22:28.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:28.369884 21680 tablet_server_runner.cc:78] Tablet server non-default flags:
18:22:28.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/logs
18:22:28.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:22:28.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
18:22:28.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:22:28.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.18.238.148:35877
18:22:28.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:22:28.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
18:22:28.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/data
18:22:28.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/wal
18:22:28.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:22:28.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:22:28.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.18.238.129:41173
18:22:28.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
18:22:28.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:22:28.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:22:28.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/data/info.pb
18:22:28.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.18.238.129
18:22:28.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44313
18:22:28.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.18.238.190:40173,127.18.238.189:35025,127.18.238.188:40453
18:22:28.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
18:22:28.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:22:28.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
18:22:28.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
18:22:28.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:22:28.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.18.238.129
18:22:28.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
18:22:28.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
18:22:28.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
18:22:28.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision cfdbd367ce7fd9747cb6f1860fc64263d9f9f79a
18:22:28.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
18:22:28.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 11 Aug 2026 18:01:34 UTC on bdcb31816ec0
18:22:28.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 17632
18:22:28.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
18:22:28.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:28.371445 21680 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
18:22:28.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:28.373034 21680 file_cache.cc:504] Constructed file cache file cache with capacity 419430
18:22:28.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:28.386821 21689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:22:30.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:29.789043 21688 debug-util.cc:398] Leaking SignalData structure 0x7b080002ede0 after lost signal to thread 21680
18:22:30.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:30.113700 21680 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.726s	user 0.522s	sys 0.986s
18:22:30.116 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:30.114176 21680 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.727s	user 0.523s	sys 0.986s
18:22:30.116 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:28.389420 21690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:22:30.117 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:30.116592 21692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:22:30.120 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:30.120090 21691 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1726 milliseconds
18:22:30.120 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:30.120170 21680 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
18:22:30.121 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:30.121371 21680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
18:22:30.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:30.123636 21680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
18:22:30.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:30.125041 21680 hybrid_clock.cc:648] HybridClock initialized: now 1786472550124986 us; error 46 us; skew 500 ppm
18:22:30.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:30.126004 21680 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
18:22:30.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:30.132663 21680 webserver.cc:533] Webserver started at http://127.18.238.129:44313/ using document root <none> and password file <none>
18:22:30.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:30.133661 21680 fs_manager.cc:362] Metadata directory not provided
18:22:30.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:30.133960 21680 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
18:22:30.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:30.142292 21680 fs_manager.cc:714] Time spent opening directory manager: real 0.005s	user 0.007s	sys 0.000s
18:22:30.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:30.147225 21708 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
18:22:30.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:30.148319 21680 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.002s
18:22:30.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:30.148751 21680 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/data,/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/wal
18:22:30.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "708d740e01414324ad249bf1854e9e5e"
18:22:30.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 18:21:47 on dist-test-slave-6nmv"
18:22:30.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:30.150772 21680 fs_report.cc:389] FS layout report
18:22:30.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
18:22:30.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/wal
18:22:30.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/wal
18:22:30.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/data/data
18:22:30.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
18:22:30.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
18:22:30.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
18:22:30.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
18:22:30.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
18:22:30.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
18:22:30.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
18:22:30.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
18:22:30.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
18:22:30.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
18:22:30.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
18:22:30.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:30.182646 21253 consensus_peers.cc:597] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 -> Peer 708d740e01414324ad249bf1854e9e5e (127.18.238.129:41173): Couldn't send request to peer 708d740e01414324ad249bf1854e9e5e. Status: Network error: Client connection negotiation failed: client connection to 127.18.238.129:41173: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
18:22:30.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:30.216946 21680 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
18:22:30.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:30.218685 21680 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
18:22:30.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:30.219106 21680 kserver.cc:163] Server-wide thread pool size limit: 3276
18:22:30.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:30.221745 21680 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
18:22:30.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:30.228556 21718 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
18:22:30.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:30.236734 21680 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
18:22:30.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:30.236943 21680 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.010s	user 0.003s	sys 0.000s
18:22:30.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:30.237151 21680 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
18:22:30.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:30.241891 21680 ts_tablet_manager.cc:616] Registered 1 tablets
18:22:30.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:30.242079 21680 ts_tablet_manager.cc:595] Time spent register tablets: real 0.005s	user 0.005s	sys 0.000s
18:22:30.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:30.242476 21718 tablet_bootstrap.cc:492] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e: Bootstrap starting.
18:22:30.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:30.308938 21718 log.cc:826] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e: Log is configured to *not* fsync() on all Append() calls
18:22:30.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:30.392831 21718 tablet_bootstrap.cc:492] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=21 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
18:22:30.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:30.395516 21718 tablet_bootstrap.cc:492] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e: Bootstrap complete.
18:22:30.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:30.398123 21718 ts_tablet_manager.cc:1403] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e: Time spent bootstrapping tablet: real 0.156s	user 0.108s	sys 0.044s
18:22:30.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:30.425670 21718 raft_consensus.cc:359] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [term 7 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:22:30.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:30.429508 21718 raft_consensus.cc:740] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: 708d740e01414324ad249bf1854e9e5e, State: Initialized, Role: FOLLOWER
18:22:30.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:30.431022 21718 consensus_queue.cc:260] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 7.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:22:30.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:30.436158 21718 ts_tablet_manager.cc:1434] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e: Time spent starting tablet: real 0.038s	user 0.033s	sys 0.003s
18:22:30.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:30.444207 21680 rpc_server.cc:307] RPC server started. Bound to: 127.18.238.129:41173
18:22:30.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:30.444589 21848 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.238.129:41173 every 8 connection(s)
18:22:30.446 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:30.446693 21680 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/data/info.pb
18:22:30.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:30.451960 19386 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu as pid 21680
18:22:30.453 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
18:22:30.474 [DEBUG - kudu-nio-5] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 24 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897, tablet=a6f8d542eda4472e9df5c193be9aa15d, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
18:22:30.476 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.18.238.130:35737
18:22:30.477 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

18:22:30.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:30.477604 19386 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu with pid 21224
18:22:30.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:30.496428 21851 heartbeater.cc:344] Connected to a master server at 127.18.238.190:40173
18:22:30.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:30.496912 21851 heartbeater.cc:461] Registering TS with master...
18:22:30.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:30.498404 21849 heartbeater.cc:344] Connected to a master server at 127.18.238.189:35025
18:22:30.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:30.498780 21849 heartbeater.cc:461] Registering TS with master...
18:22:30.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:30.498715 21851 heartbeater.cc:507] Master 127.18.238.190:40173 requested a full tablet report, sending...
18:22:30.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:30.499775 21849 heartbeater.cc:507] Master 127.18.238.189:35025 requested a full tablet report, sending...
18:22:30.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:30.503278 21850 heartbeater.cc:344] Connected to a master server at 127.18.238.188:40453
18:22:30.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:30.503590 21850 heartbeater.cc:461] Registering TS with master...
18:22:30.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:30.504506 21850 heartbeater.cc:507] Master 127.18.238.188:40453 requested a full tablet report, sending...
18:22:30.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:30.504460 19440 ts_manager.cc:194] Re-registered known tserver with Master: 708d740e01414324ad249bf1854e9e5e (127.18.238.129:41173)
18:22:30.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:30.504566 19526 ts_manager.cc:194] Re-registered known tserver with Master: 708d740e01414324ad249bf1854e9e5e (127.18.238.129:41173)
18:22:30.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:30.507524 19624 ts_manager.cc:194] Re-registered known tserver with Master: 708d740e01414324ad249bf1854e9e5e (127.18.238.129:41173)
18:22:30.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:30.509496 19624 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.18.238.129:44607
18:22:30.525 [DEBUG - kudu-nio-5] (Connection.java:239) [peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)] handling channelInactive
18:22:30.525 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)] cleaning up while in state READY due to: connection closed
18:22:30.525 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
18:22:30.526 [DEBUG - Time-limited test] (Connection.java:748) [peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)] connecting to peer
18:22:30.527 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737
18:22:30.528 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)] cleaning up while in state CONNECTING due to: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737
18:22:30.528 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737) for tablet a6f8d542eda4472e9df5c193be9aa15d: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737
18:22:30.538 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:22:30.538 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)[L],489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557),708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)]}]
18:22:30.538 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=a6f8d542eda4472e9df5c193be9aa15d, attempt=2, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=12), Traces: [0ms] sending RPC to server 24001d6c96764c04999eb5de15b9eba9, [3ms] received response from server 24001d6c96764c04999eb5de15b9eba9: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [3ms] delaying RPC due to: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [12ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=Deferred@404698668(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007f5c7c36b7a8@308a3517 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@12859e83 -> wakeup thread Time-limited test)) after lookup
18:22:30.538 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)] connecting to peer
18:22:30.539 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737
18:22:30.539 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)] cleaning up while in state CONNECTING due to: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737
18:22:30.539 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737) for tablet a6f8d542eda4472e9df5c193be9aa15d: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737
18:22:30.557 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:22:30.557 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)[L],489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557),708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)]}]
18:22:30.558 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=a6f8d542eda4472e9df5c193be9aa15d, attempt=4, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=32), Traces: [0ms] sending RPC to server 24001d6c96764c04999eb5de15b9eba9, [3ms] received response from server 24001d6c96764c04999eb5de15b9eba9: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [3ms] delaying RPC due to: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [12ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [13ms] sending RPC to server 24001d6c96764c04999eb5de15b9eba9, [14ms] received response from server 24001d6c96764c04999eb5de15b9eba9: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [14ms] delaying RPC due to: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [32ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=Deferred@404698668(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007f5c7c36b7a8@308a3517 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@436640910 after retry RPC@617183435), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@12859e83 -> wakeup thread Time-limited test -> (continuation of Deferred@436640910 after retry RPC@617183435))) after lookup
18:22:30.558 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)] connecting to peer
18:22:30.559 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737
18:22:30.559 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)] cleaning up while in state CONNECTING due to: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737
18:22:30.560 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737) for tablet a6f8d542eda4472e9df5c193be9aa15d: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737
18:22:30.578 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:22:30.578 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)[L],489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557),708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)]}]
18:22:30.579 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=a6f8d542eda4472e9df5c193be9aa15d, attempt=6, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=53), Traces: [0ms] sending RPC to server 24001d6c96764c04999eb5de15b9eba9, [3ms] received response from server 24001d6c96764c04999eb5de15b9eba9: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [3ms] delaying RPC due to: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [12ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [13ms] sending RPC to server 24001d6c96764c04999eb5de15b9eba9, [14ms] received response from server 24001d6c96764c04999eb5de15b9eba9: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [14ms] delaying RPC due to: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [32ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [33ms] sending RPC to server 24001d6c96764c04999eb5de15b9eba9, [34ms] received response from server 24001d6c96764c04999eb5de15b9eba9: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [35ms] delaying RPC due to: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [53ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=Deferred@404698668(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007f5c7c36b7a8@308a3517 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@436640910 after retry RPC@617183435) -> (continuation of Deferred@336416246 after retry RPC@2070769820), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@12859e83 -> wakeup thread Time-limited test -> (continuation of Deferred@436640910 after retry RPC@617183435) -> (continuation of Deferred@336416246 after retry RPC@2070769820))) after lookup
18:22:30.579 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)] connecting to peer
18:22:30.580 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737
18:22:30.580 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)] cleaning up while in state CONNECTING due to: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737
18:22:30.581 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737) for tablet a6f8d542eda4472e9df5c193be9aa15d: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737
18:22:30.699 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:22:30.699 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)[L],489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557),708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)]}]
18:22:30.700 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=a6f8d542eda4472e9df5c193be9aa15d, attempt=8, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=174), Traces: [0ms] sending RPC to server 24001d6c96764c04999eb5de15b9eba9, [3ms] received response from server 24001d6c96764c04999eb5de15b9eba9: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [3ms] delaying RPC due to: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [12ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [13ms] sending RPC to server 24001d6c96764c04999eb5de15b9eba9, [14ms] received response from server 24001d6c96764c04999eb5de15b9eba9: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [14ms] delaying RPC due to: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [32ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [33ms] sending RPC to server 24001d6c96764c04999eb5de15b9eba9, [34ms] received response from server 24001d6c96764c04999eb5de15b9eba9: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [35ms] delaying RPC due to: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [53ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [54ms] sending RPC to server 24001d6c96764c04999eb5de15b9eba9, [56ms] received response from server 24001d6c96764c04999eb5de15b9eba9: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [56ms] delaying RPC due to: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [170ms] refreshing cache from master, [170ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [174ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=Deferred@404698668(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007f5c7c36b7a8@308a3517 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@436640910 after retry RPC@617183435) -> (continuation of Deferred@336416246 after retry RPC@2070769820) -> (continuation of Deferred@1892513937 after retry RPC@1622413188), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@12859e83 -> wakeup thread Time-limited test -> (continuation of Deferred@436640910 after retry RPC@617183435) -> (continuation of Deferred@336416246 after retry RPC@2070769820) -> (continuation of Deferred@1892513937 after retry RPC@1622413188))) after lookup
18:22:30.700 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)] connecting to peer
18:22:30.701 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737
18:22:30.702 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)] cleaning up while in state CONNECTING due to: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737
18:22:30.702 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737) for tablet a6f8d542eda4472e9df5c193be9aa15d: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737
18:22:30.958 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:22:30.958 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)[L],489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557),708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)]}]
18:22:30.958 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=a6f8d542eda4472e9df5c193be9aa15d, attempt=10, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=433), Traces: [0ms] sending RPC to server 24001d6c96764c04999eb5de15b9eba9, [3ms] received response from server 24001d6c96764c04999eb5de15b9eba9: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [3ms] delaying RPC due to: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [12ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [13ms] sending RPC to server 24001d6c96764c04999eb5de15b9eba9, [14ms] received response from server 24001d6c96764c04999eb5de15b9eba9: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [14ms] delaying RPC due to: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [32ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [33ms] sending RPC to server 24001d6c96764c04999eb5de15b9eba9, [34ms] received response from server 24001d6c96764c04999eb5de15b9eba9: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [35ms] delaying RPC due to: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [53ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [54ms] sending RPC to server 24001d6c96764c04999eb5de15b9eba9, [56ms] received response from server 24001d6c96764c04999eb5de15b9eba9: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [56ms] delaying RPC due to: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [170ms] refreshing cache from master, [170ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [174ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [175ms] sending RPC to server 24001d6c96764c04999eb5de15b9eba9, [177ms] received response from server 24001d6c96764c04999eb5de15b9eba9: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [177ms] delaying RPC due to: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [429ms] refreshing cache from master, [429ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [432ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=Deferred@404698668(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007f5c7c36b7a8@308a3517 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@436640910 after retry RPC@617183435) -> (continuation of Deferred@336416246 after retry RPC@2070769820) -> (continuation of Deferred@1892513937 after retry RPC@1622413188) -> (continuation of Deferred@155478369 after retry RPC@543377074), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@12859e83 -> wakeup thread Time-limited test -> (continuation of Deferred@436640910 after retry RPC@617183435) -> (continuation of Deferred@336416246 after retry RPC@2070769820) -> (continuation of Deferred@1892513937 after retry RPC@1622413188) -> (continuation of Deferred@155478369 after retry RPC@543377074))) after lookup
18:22:30.959 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)] connecting to peer
18:22:30.960 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737
18:22:30.960 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)] cleaning up while in state CONNECTING due to: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737
18:22:30.960 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737) for tablet a6f8d542eda4472e9df5c193be9aa15d: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737
18:22:31.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:31.417593 21864 raft_consensus.cc:493] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 24001d6c96764c04999eb5de15b9eba9)
18:22:31.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:31.418033 21864 raft_consensus.cc:515] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:22:31.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:31.419452 21864 leader_election.cc:290] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 24001d6c96764c04999eb5de15b9eba9 (127.18.238.130:35737), 708d740e01414324ad249bf1854e9e5e (127.18.238.129:41173)
18:22:31.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:31.425781 21492 leader_election.cc:336] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 24001d6c96764c04999eb5de15b9eba9 (127.18.238.130:35737): Network error: Client connection negotiation failed: client connection to 127.18.238.130:35737: connect: Connection refused (error 111)
18:22:31.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:31.431522 21795 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6f8d542eda4472e9df5c193be9aa15d" candidate_uuid: "489d187f67e24039b9c5ea35f7d1e11e" candidate_term: 9 candidate_status { last_received { term: 8 index: 32 } } ignore_live_leader: false dest_uuid: "708d740e01414324ad249bf1854e9e5e" is_pre_election: true
18:22:31.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:31.432224 21795 raft_consensus.cc:2468] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 489d187f67e24039b9c5ea35f7d1e11e in term 7.
18:22:31.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:31.433382 21492 leader_election.cc:304] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 489d187f67e24039b9c5ea35f7d1e11e, 708d740e01414324ad249bf1854e9e5e; no voters: 24001d6c96764c04999eb5de15b9eba9
18:22:31.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:31.434075 21864 raft_consensus.cc:2804] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 8 FOLLOWER]: Leader pre-election won for term 9
18:22:31.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:31.434322 21864 raft_consensus.cc:493] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 8 FOLLOWER]: Starting leader election (detected failure of leader 24001d6c96764c04999eb5de15b9eba9)
18:22:31.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:31.434551 21864 raft_consensus.cc:3060] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 8 FOLLOWER]: Advancing to term 9
18:22:31.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:31.439020 21864 raft_consensus.cc:515] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:22:31.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:31.440374 21864 leader_election.cc:290] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [CANDIDATE]: Term 9 election: Requested vote from peers 24001d6c96764c04999eb5de15b9eba9 (127.18.238.130:35737), 708d740e01414324ad249bf1854e9e5e (127.18.238.129:41173)
18:22:31.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:31.442108 21795 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6f8d542eda4472e9df5c193be9aa15d" candidate_uuid: "489d187f67e24039b9c5ea35f7d1e11e" candidate_term: 9 candidate_status { last_received { term: 8 index: 32 } } ignore_live_leader: false dest_uuid: "708d740e01414324ad249bf1854e9e5e"
18:22:31.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:31.442705 21795 raft_consensus.cc:3060] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [term 7 FOLLOWER]: Advancing to term 9
18:22:31.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:31.444100 21492 leader_election.cc:336] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 24001d6c96764c04999eb5de15b9eba9 (127.18.238.130:35737): Network error: Client connection negotiation failed: client connection to 127.18.238.130:35737: connect: Connection refused (error 111)
18:22:31.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:31.452064 21795 raft_consensus.cc:2468] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 489d187f67e24039b9c5ea35f7d1e11e in term 9.
18:22:31.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:31.453271 21492 leader_election.cc:304] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 489d187f67e24039b9c5ea35f7d1e11e, 708d740e01414324ad249bf1854e9e5e; no voters: 24001d6c96764c04999eb5de15b9eba9
18:22:31.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:31.454037 21864 raft_consensus.cc:2804] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 9 FOLLOWER]: Leader election won for term 9
18:22:31.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:31.455367 21864 raft_consensus.cc:697] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 9 LEADER]: Becoming Leader. State: Replica: 489d187f67e24039b9c5ea35f7d1e11e, State: Running, Role: LEADER
18:22:31.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:31.456508 21864 consensus_queue.cc:237] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 8.32, Last appended by leader: 32, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:22:31.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:31.466499 19624 catalog_manager.cc:5719] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e reported cstate change: term changed from 8 to 9, leader changed from 24001d6c96764c04999eb5de15b9eba9 (127.18.238.130) to 489d187f67e24039b9c5ea35f7d1e11e (127.18.238.131). New cstate: current_term: 9 leader_uuid: "489d187f67e24039b9c5ea35f7d1e11e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } health_report { overall_health: UNKNOWN } } }
18:22:31.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:31.512248 21850 heartbeater.cc:499] Master 127.18.238.188:40453 was elected leader, sending a full tablet report...
18:22:31.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:31.914979 21492 consensus_peers.cc:597] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e -> Peer 24001d6c96764c04999eb5de15b9eba9 (127.18.238.130:35737): Couldn't send request to peer 24001d6c96764c04999eb5de15b9eba9. Status: Network error: Client connection negotiation failed: client connection to 127.18.238.130:35737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
18:22:32.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:32.049935 19561 debug-util.cc:398] Leaking SignalData structure 0x7b0800059780 after lost signal to thread 19488
18:22:32.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:32.050586 19561 debug-util.cc:398] Leaking SignalData structure 0x7b0800059180 after lost signal to thread 19566
18:22:32.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:32.053733 21795 raft_consensus.cc:1275] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [term 9 FOLLOWER]: Refusing update from remote peer 489d187f67e24039b9c5ea35f7d1e11e: Log matching property violated. Preceding OpId in replica: term: 7 index: 28. Preceding OpId from leader: term: 9 index: 33. (index mismatch)
18:22:32.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:32.055316 21864 consensus_queue.cc:1048] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [LEADER]: Connected to new peer: Peer: permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 28, Time since last communication: 0.000s
18:22:32.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:32.461302 21492 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.238.130:35737: connect: Connection refused (error 111) [suppressed 3 similar messages]
18:22:32.499 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:22:32.499 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737),489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)[L],708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)]}]
18:22:32.500 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=a6f8d542eda4472e9df5c193be9aa15d, attempt=12, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=1974), Traces: [0ms] sending RPC to server 24001d6c96764c04999eb5de15b9eba9, [3ms] received response from server 24001d6c96764c04999eb5de15b9eba9: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [3ms] delaying RPC due to: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [12ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [13ms] sending RPC to server 24001d6c96764c04999eb5de15b9eba9, [14ms] received response from server 24001d6c96764c04999eb5de15b9eba9: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [14ms] delaying RPC due to: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [32ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [33ms] sending RPC to server 24001d6c96764c04999eb5de15b9eba9, [34ms] received response from server 24001d6c96764c04999eb5de15b9eba9: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [35ms] delaying RPC due to: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [53ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [54ms] sending RPC to server 24001d6c96764c04999eb5de15b9eba9, [56ms] received response from server 24001d6c96764c04999eb5de15b9eba9: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [56ms] delaying RPC due to: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [170ms] refreshing cache from master, [170ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [174ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [175ms] sending RPC to server 24001d6c96764c04999eb5de15b9eba9, [177ms] received response from server 24001d6c96764c04999eb5de15b9eba9: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [177ms] delaying RPC due to: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [429ms] refreshing cache from master, [429ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [432ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [434ms] sending RPC to server 24001d6c96764c04999eb5de15b9eba9, [435ms] received response from server 24001d6c96764c04999eb5de15b9eba9: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [435ms] delaying RPC due to: Network error: Failed to connect to peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737): Connection refused: /127.18.238.130:35737, [1969ms] refreshing cache from master, [1969ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [1974ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=Deferred@404698668(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007f5c7c36b7a8@308a3517 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@436640910 after retry RPC@617183435) -> (continuation of Deferred@336416246 after retry RPC@2070769820) -> (continuation of Deferred@1892513937 after retry RPC@1622413188) -> (continuation of Deferred@155478369 after retry RPC@543377074) -> (continuation of Deferred@226595228 after retry RPC@938283080), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@12859e83 -> wakeup thread Time-limited test -> (continuation of Deferred@436640910 after retry RPC@617183435) -> (continuation of Deferred@336416246 after retry RPC@2070769820) -> (continuation of Deferred@1892513937 after retry RPC@1622413188) -> (continuation of Deferred@155478369 after retry RPC@543377074) -> (continuation of Deferred@226595228 after retry RPC@938283080))) after lookup
18:22:32.500 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)] connecting to peer
18:22:32.501 [DEBUG - kudu-nio-4] (Connection.java:757) [peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)] Successfully connected to peer
18:22:32.528 [DEBUG - kudu-nio-4] (Negotiator.java:918) Authenticated connection [id: 0x0f4f587e, L:/127.0.0.1:47678 - R:/127.18.238.131:46557] using TOKEN/null
18:22:32.570 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.18.238.130:35737
18:22:32.570 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

18:22:32.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:32.572266 19386 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu
18:22:32.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu
18:22:32.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-1/wal
18:22:32.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-1/data
18:22:32.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
18:22:32.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
18:22:32.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
18:22:32.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:22:32.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:22:32.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:22:32.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-1/logs
18:22:32.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-1/data/info.pb
18:22:32.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:22:32.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
18:22:32.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
18:22:32.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
18:22:32.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
18:22:32.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:22:32.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:22:32.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:22:32.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
18:22:32.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
18:22:32.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.18.238.130:35737
18:22:32.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.18.238.130
18:22:32.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.18.238.190:40173,127.18.238.189:35025,127.18.238.188:40453
18:22:32.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44475
18:22:32.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.18.238.130
18:22:32.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.18.238.148:35877
18:22:32.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:22:32.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:22:32.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
18:22:32.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:32.920197 21901 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
18:22:32.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:32.920779 21901 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
18:22:32.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:32.921378 21901 flags.cc:432] Enabled unsafe flag: --never_fsync=true
18:22:32.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:32.954018 21901 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
18:22:32.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:32.954921 21901 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.238.130
18:22:32.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:32.988390 21901 tablet_server_runner.cc:78] Tablet server non-default flags:
18:22:32.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-1/logs
18:22:32.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:22:32.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
18:22:32.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:22:32.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.18.238.148:35877
18:22:32.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:22:32.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
18:22:32.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-1/data
18:22:32.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-1/wal
18:22:32.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:22:32.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:22:32.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.18.238.130:35737
18:22:32.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
18:22:32.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:22:32.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:22:32.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-1/data/info.pb
18:22:32.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.18.238.130
18:22:32.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44475
18:22:32.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.18.238.190:40173,127.18.238.189:35025,127.18.238.188:40453
18:22:32.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
18:22:32.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:22:32.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
18:22:32.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
18:22:32.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:22:32.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.18.238.130
18:22:32.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
18:22:32.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
18:22:32.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
18:22:32.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision cfdbd367ce7fd9747cb6f1860fc64263d9f9f79a
18:22:32.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
18:22:32.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 11 Aug 2026 18:01:34 UTC on bdcb31816ec0
18:22:32.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 17632
18:22:32.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
18:22:32.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:32.989992 21901 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
18:22:32.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:32.991463 21901 file_cache.cc:504] Constructed file cache file cache with capacity 419430
18:22:33.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:33.004128 21910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:22:34.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:33.004796 21911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:22:34.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:34.365545 21913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:22:34.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:34.368603 21901 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.364s	user 0.459s	sys 0.881s
18:22:34.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:34.369158 21901 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.365s	user 0.460s	sys 0.883s
18:22:34.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:34.371049 21912 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1365 milliseconds
18:22:34.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:34.371143 21901 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
18:22:34.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:34.372821 21901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
18:22:34.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:34.375599 21901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
18:22:34.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:34.377065 21901 hybrid_clock.cc:648] HybridClock initialized: now 1786472554377005 us; error 50 us; skew 500 ppm
18:22:34.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:34.378265 21901 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
18:22:34.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:34.387342 21901 webserver.cc:533] Webserver started at http://127.18.238.130:44475/ using document root <none> and password file <none>
18:22:34.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:34.388629 21901 fs_manager.cc:362] Metadata directory not provided
18:22:34.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:34.388902 21901 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
18:22:34.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:34.399407 21901 fs_manager.cc:714] Time spent opening directory manager: real 0.006s	user 0.007s	sys 0.001s
18:22:34.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:34.404904 21923 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
18:22:34.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:34.406160 21901 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.003s	sys 0.003s
18:22:34.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:34.406515 21901 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-1/data,/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-1/wal
18:22:34.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "24001d6c96764c04999eb5de15b9eba9"
18:22:34.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 18:21:50 on dist-test-slave-6nmv"
18:22:34.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:34.408383 21901 fs_report.cc:389] FS layout report
18:22:34.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
18:22:34.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-1/wal
18:22:34.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-1/wal
18:22:34.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-1/data/data
18:22:34.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
18:22:34.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
18:22:34.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
18:22:34.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
18:22:34.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
18:22:34.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
18:22:34.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
18:22:34.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
18:22:34.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
18:22:34.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
18:22:34.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
18:22:34.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:34.486917 21901 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
18:22:34.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:34.488463 21901 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
18:22:34.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:34.488897 21901 kserver.cc:163] Server-wide thread pool size limit: 3276
18:22:34.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:34.491911 21901 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
18:22:34.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:34.498302 21932 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
18:22:34.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:34.506224 21901 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
18:22:34.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:34.506421 21901 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.010s	user 0.002s	sys 0.000s
18:22:34.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:34.506656 21901 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
18:22:34.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:34.513608 21901 ts_tablet_manager.cc:616] Registered 1 tablets
18:22:34.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:34.513863 21901 ts_tablet_manager.cc:595] Time spent register tablets: real 0.007s	user 0.008s	sys 0.000s
18:22:34.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:34.514353 21932 tablet_bootstrap.cc:492] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9: Bootstrap starting.
18:22:34.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:34.588608 21932 log.cc:826] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9: Log is configured to *not* fsync() on all Append() calls
18:22:34.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:34.696349 21901 rpc_server.cc:307] RPC server started. Bound to: 127.18.238.130:35737
18:22:34.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:34.696892 22050 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.238.130:35737 every 8 connection(s)
18:22:34.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:34.699885 21901 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-1/data/info.pb
18:22:34.705 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
18:22:34.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:34.704555 19386 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu as pid 21901
18:22:34.762 [DEBUG - kudu-nio-4] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 27 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897, tablet=a6f8d542eda4472e9df5c193be9aa15d, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
18:22:34.763 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.18.238.131:46557
18:22:34.763 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

18:22:34.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:34.749653 21932 tablet_bootstrap.cc:492] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=24 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
18:22:34.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:34.764114 19386 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu with pid 21463
18:22:34.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:34.764657 21932 tablet_bootstrap.cc:492] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9: Bootstrap complete.
18:22:34.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:34.767886 21932 ts_tablet_manager.cc:1403] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9: Time spent bootstrapping tablet: real 0.254s	user 0.143s	sys 0.066s
18:22:34.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:34.793177 22051 heartbeater.cc:344] Connected to a master server at 127.18.238.189:35025
18:22:34.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:34.793565 22051 heartbeater.cc:461] Registering TS with master...
18:22:34.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:34.794827 22051 heartbeater.cc:507] Master 127.18.238.189:35025 requested a full tablet report, sending...
18:22:34.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:34.797111 22056 heartbeater.cc:344] Connected to a master server at 127.18.238.190:40173
18:22:34.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:34.797370 22056 heartbeater.cc:461] Registering TS with master...
18:22:34.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:34.798017 22056 heartbeater.cc:507] Master 127.18.238.190:40173 requested a full tablet report, sending...
18:22:34.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:34.799966 19524 ts_manager.cc:194] Re-registered known tserver with Master: 24001d6c96764c04999eb5de15b9eba9 (127.18.238.130:35737)
18:22:34.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:34.802865 19441 ts_manager.cc:194] Re-registered known tserver with Master: 24001d6c96764c04999eb5de15b9eba9 (127.18.238.130:35737)
18:22:34.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:34.809482 22052 heartbeater.cc:344] Connected to a master server at 127.18.238.188:40453
18:22:34.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:34.809846 22052 heartbeater.cc:461] Registering TS with master...
18:22:34.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:34.810583 22052 heartbeater.cc:507] Master 127.18.238.188:40453 requested a full tablet report, sending...
18:22:34.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:34.813179 19624 ts_manager.cc:194] Re-registered known tserver with Master: 24001d6c96764c04999eb5de15b9eba9 (127.18.238.130:35737)
18:22:34.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:34.816262 19624 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.18.238.130:36823
18:22:34.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:34.815653 21932 raft_consensus.cc:359] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 8 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:22:34.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:34.818403 21932 raft_consensus.cc:740] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24001d6c96764c04999eb5de15b9eba9, State: Initialized, Role: FOLLOWER
18:22:34.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:34.819175 21932 consensus_queue.cc:260] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 8.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:22:34.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:34.821797 22052 heartbeater.cc:499] Master 127.18.238.188:40453 was elected leader, sending a full tablet report...
18:22:34.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:34.822183 21932 ts_tablet_manager.cc:1434] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9: Time spent starting tablet: real 0.054s	user 0.024s	sys 0.001s
18:22:34.823 [DEBUG - kudu-nio-4] (Connection.java:239) [peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)] handling channelInactive
18:22:34.823 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)] cleaning up while in state READY due to: connection closed
18:22:34.826 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
18:22:34.827 [DEBUG - Time-limited test] (Connection.java:748) [peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)] connecting to peer
18:22:34.828 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557
18:22:34.828 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)] cleaning up while in state CONNECTING due to: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557
18:22:34.828 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557) for tablet a6f8d542eda4472e9df5c193be9aa15d: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557
18:22:34.838 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:22:34.838 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737),489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)[L],708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)]}]
18:22:34.838 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=a6f8d542eda4472e9df5c193be9aa15d, attempt=2, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=12), Traces: [0ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [2ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [2ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [12ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=Deferred@1290078442(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007f5c7c36b7a8@13d53e6e -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6b3d44a7 -> wakeup thread Time-limited test)) after lookup
18:22:34.839 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)] connecting to peer
18:22:34.840 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557
18:22:34.840 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)] cleaning up while in state CONNECTING due to: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557
18:22:34.840 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557) for tablet a6f8d542eda4472e9df5c193be9aa15d: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557
18:22:34.857 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:22:34.857 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737),489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)[L],708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)]}]
18:22:34.857 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=a6f8d542eda4472e9df5c193be9aa15d, attempt=4, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=31), Traces: [0ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [2ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [2ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [12ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [13ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [14ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [14ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [28ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [31ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=Deferred@1290078442(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007f5c7c36b7a8@13d53e6e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@862285533 after retry RPC@846990442), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6b3d44a7 -> wakeup thread Time-limited test -> (continuation of Deferred@862285533 after retry RPC@846990442))) after lookup
18:22:34.858 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)] connecting to peer
18:22:34.859 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557
18:22:34.859 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)] cleaning up while in state CONNECTING due to: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557
18:22:34.859 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557) for tablet a6f8d542eda4472e9df5c193be9aa15d: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557
18:22:34.877 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:22:34.878 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737),489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)[L],708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)]}]
18:22:34.878 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=a6f8d542eda4472e9df5c193be9aa15d, attempt=6, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=51), Traces: [0ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [2ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [2ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [12ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [13ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [14ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [14ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [28ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [31ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [32ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [33ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [33ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [48ms] refreshing cache from master, [48ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [51ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=Deferred@1290078442(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007f5c7c36b7a8@13d53e6e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@862285533 after retry RPC@846990442) -> (continuation of Deferred@2044339716 after retry RPC@722954761), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6b3d44a7 -> wakeup thread Time-limited test -> (continuation of Deferred@862285533 after retry RPC@846990442) -> (continuation of Deferred@2044339716 after retry RPC@722954761))) after lookup
18:22:34.879 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)] connecting to peer
18:22:34.880 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557
18:22:34.880 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)] cleaning up while in state CONNECTING due to: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557
18:22:34.880 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557) for tablet a6f8d542eda4472e9df5c193be9aa15d: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557
18:22:34.958 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:22:34.959 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737),489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)[L],708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)]}]
18:22:34.959 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=a6f8d542eda4472e9df5c193be9aa15d, attempt=8, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=132), Traces: [0ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [2ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [2ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [12ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [13ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [14ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [14ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [28ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [31ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [32ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [33ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [33ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [48ms] refreshing cache from master, [48ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [51ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [52ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [54ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [54ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [128ms] refreshing cache from master, [128ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [132ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=Deferred@1290078442(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007f5c7c36b7a8@13d53e6e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@862285533 after retry RPC@846990442) -> (continuation of Deferred@2044339716 after retry RPC@722954761) -> (continuation of Deferred@1879446162 after retry RPC@1615952313), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6b3d44a7 -> wakeup thread Time-limited test -> (continuation of Deferred@862285533 after retry RPC@846990442) -> (continuation of Deferred@2044339716 after retry RPC@722954761) -> (continuation of Deferred@1879446162 after retry RPC@1615952313))) after lookup
18:22:34.960 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)] connecting to peer
18:22:34.960 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557
18:22:34.961 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)] cleaning up while in state CONNECTING due to: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557
18:22:34.961 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557) for tablet a6f8d542eda4472e9df5c193be9aa15d: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557
18:22:35.358 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:22:35.359 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737),489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)[L],708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)]}]
18:22:35.360 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=a6f8d542eda4472e9df5c193be9aa15d, attempt=10, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=533), Traces: [0ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [2ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [2ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [12ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [13ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [14ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [14ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [28ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [31ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [32ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [33ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [33ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [48ms] refreshing cache from master, [48ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [51ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [52ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [54ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [54ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [128ms] refreshing cache from master, [128ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [132ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [133ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [135ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [135ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [528ms] refreshing cache from master, [528ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [532ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=Deferred@1290078442(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007f5c7c36b7a8@13d53e6e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@862285533 after retry RPC@846990442) -> (continuation of Deferred@2044339716 after retry RPC@722954761) -> (continuation of Deferred@1879446162 after retry RPC@1615952313) -> (continuation of Deferred@2034748568 after retry RPC@305200824), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6b3d44a7 -> wakeup thread Time-limited test -> (continuation of Deferred@862285533 after retry RPC@846990442) -> (continuation of Deferred@2044339716 after retry RPC@722954761) -> (continuation of Deferred@1879446162 after retry RPC@1615952313) -> (continuation of Deferred@2034748568 after retry RPC@305200824))) after lookup
18:22:35.361 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)] connecting to peer
18:22:35.362 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557
18:22:35.362 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)] cleaning up while in state CONNECTING due to: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557
18:22:35.362 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557) for tablet a6f8d542eda4472e9df5c193be9aa15d: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557
18:22:35.899 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:22:35.899 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737),489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)[L],708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)]}]
18:22:35.900 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=a6f8d542eda4472e9df5c193be9aa15d, attempt=12, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=1073), Traces: [0ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [2ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [2ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [12ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [13ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [14ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [14ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [28ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [31ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [32ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [33ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [33ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [48ms] refreshing cache from master, [48ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [51ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [52ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [54ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [54ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [128ms] refreshing cache from master, [128ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [132ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [133ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [135ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [135ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [528ms] refreshing cache from master, [528ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [532ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [535ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [536ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [536ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [1068ms] refreshing cache from master, [1068ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [1073ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=Deferred@1290078442(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007f5c7c36b7a8@13d53e6e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@862285533 after retry RPC@846990442) -> (continuation of Deferred@2044339716 after retry RPC@722954761) -> (continuation of Deferred@1879446162 after retry RPC@1615952313) -> (continuation of Deferred@2034748568 after retry RPC@305200824) -> (continuation of Deferred@1953402276 after retry RPC@1821832832), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6b3d44a7 -> wakeup thread Time-limited test -> (continuation of Deferred@862285533 after retry RPC@846990442) -> (continuation of Deferred@2044339716 after retry RPC@722954761) -> (continuation of Deferred@1879446162 after retry RPC@1615952313) -> (continuation of Deferred@2034748568 after retry RPC@305200824) -> (continuation of Deferred@1953402276 after retry RPC@1821832832))) after lookup
18:22:35.900 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)] connecting to peer
18:22:35.901 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557
18:22:35.902 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)] cleaning up while in state CONNECTING due to: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557
18:22:35.902 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557) for tablet a6f8d542eda4472e9df5c193be9aa15d: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557
18:22:36.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:36.167302 22068 raft_consensus.cc:493] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 489d187f67e24039b9c5ea35f7d1e11e)
18:22:36.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:36.167766 22068 raft_consensus.cc:515] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:22:36.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:36.170270 22068 leader_election.cc:290] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 24001d6c96764c04999eb5de15b9eba9 (127.18.238.130:35737), 489d187f67e24039b9c5ea35f7d1e11e (127.18.238.131:46557)
18:22:36.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:36.175302 21712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.238.131:46557: connect: Connection refused (error 111)
18:22:36.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:36.179533 21712 leader_election.cc:336] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 489d187f67e24039b9c5ea35f7d1e11e (127.18.238.131:46557): Network error: Client connection negotiation failed: client connection to 127.18.238.131:46557: connect: Connection refused (error 111)
18:22:36.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:36.182276 22002 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6f8d542eda4472e9df5c193be9aa15d" candidate_uuid: "708d740e01414324ad249bf1854e9e5e" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "24001d6c96764c04999eb5de15b9eba9" is_pre_election: true
18:22:36.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:36.183020 22002 raft_consensus.cc:2468] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 708d740e01414324ad249bf1854e9e5e in term 8.
18:22:36.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:36.184413 21714 leader_election.cc:304] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24001d6c96764c04999eb5de15b9eba9, 708d740e01414324ad249bf1854e9e5e; no voters: 489d187f67e24039b9c5ea35f7d1e11e
18:22:36.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:36.185094 22068 raft_consensus.cc:2804] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [term 9 FOLLOWER]: Leader pre-election won for term 10
18:22:36.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:36.185401 22068 raft_consensus.cc:493] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [term 9 FOLLOWER]: Starting leader election (detected failure of leader 489d187f67e24039b9c5ea35f7d1e11e)
18:22:36.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:36.185652 22068 raft_consensus.cc:3060] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [term 9 FOLLOWER]: Advancing to term 10
18:22:36.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:36.189988 22068 raft_consensus.cc:515] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:22:36.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:36.191432 22068 leader_election.cc:290] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [CANDIDATE]: Term 10 election: Requested vote from peers 24001d6c96764c04999eb5de15b9eba9 (127.18.238.130:35737), 489d187f67e24039b9c5ea35f7d1e11e (127.18.238.131:46557)
18:22:36.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:36.192201 22002 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6f8d542eda4472e9df5c193be9aa15d" candidate_uuid: "708d740e01414324ad249bf1854e9e5e" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "24001d6c96764c04999eb5de15b9eba9"
18:22:36.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:36.192737 22002 raft_consensus.cc:3060] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 8 FOLLOWER]: Advancing to term 10
18:22:36.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:36.195756 21712 leader_election.cc:336] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 489d187f67e24039b9c5ea35f7d1e11e (127.18.238.131:46557): Network error: Client connection negotiation failed: client connection to 127.18.238.131:46557: connect: Connection refused (error 111)
18:22:36.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:36.199457 22002 raft_consensus.cc:2468] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 708d740e01414324ad249bf1854e9e5e in term 10.
18:22:36.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:36.200430 21714 leader_election.cc:304] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24001d6c96764c04999eb5de15b9eba9, 708d740e01414324ad249bf1854e9e5e; no voters: 489d187f67e24039b9c5ea35f7d1e11e
18:22:36.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:36.201031 22068 raft_consensus.cc:2804] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [term 10 FOLLOWER]: Leader election won for term 10
18:22:36.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:36.202663 22068 raft_consensus.cc:697] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [term 10 LEADER]: Becoming Leader. State: Replica: 708d740e01414324ad249bf1854e9e5e, State: Running, Role: LEADER
18:22:36.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:36.203647 22068 consensus_queue.cc:237] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 9.36, Last appended by leader: 36, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:22:36.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:36.211897 19624 catalog_manager.cc:5719] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e reported cstate change: term changed from 9 to 10, leader changed from 489d187f67e24039b9c5ea35f7d1e11e (127.18.238.131) to 708d740e01414324ad249bf1854e9e5e (127.18.238.129). New cstate: current_term: 10 leader_uuid: "708d740e01414324ad249bf1854e9e5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } health_report { overall_health: HEALTHY } } }
18:22:36.298 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:22:36.299 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737),489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557),708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)[L]]}]
18:22:36.299 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=a6f8d542eda4472e9df5c193be9aa15d, attempt=14, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=1472), Traces: [0ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [2ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [2ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [12ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [13ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [14ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [14ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [28ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [31ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [32ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [33ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [33ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [48ms] refreshing cache from master, [48ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [51ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [52ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [54ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [54ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [128ms] refreshing cache from master, [128ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [132ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [133ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [135ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [135ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [528ms] refreshing cache from master, [528ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [532ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [535ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [536ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [536ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [1068ms] refreshing cache from master, [1068ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [1073ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [1074ms] sending RPC to server 489d187f67e24039b9c5ea35f7d1e11e, [1076ms] received response from server 489d187f67e24039b9c5ea35f7d1e11e: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [1076ms] delaying RPC due to: Network error: Failed to connect to peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557): Connection refused: /127.18.238.131:46557, [1469ms] refreshing cache from master, [1469ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [1472ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=Deferred@1290078442(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007f5c7c36b7a8@13d53e6e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@862285533 after retry RPC@846990442) -> (continuation of Deferred@2044339716 after retry RPC@722954761) -> (continuation of Deferred@1879446162 after retry RPC@1615952313) -> (continuation of Deferred@2034748568 after retry RPC@305200824) -> (continuation of Deferred@1953402276 after retry RPC@1821832832) -> (continuation of Deferred@2115745266 after retry RPC@627207751), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6b3d44a7 -> wakeup thread Time-limited test -> (continuation of Deferred@862285533 after retry RPC@846990442) -> (continuation of Deferred@2044339716 after retry RPC@722954761) -> (continuation of Deferred@1879446162 after retry RPC@1615952313) -> (continuation of Deferred@2034748568 after retry RPC@305200824) -> (continuation of Deferred@1953402276 after retry RPC@1821832832) -> (continuation of Deferred@2115745266 after retry RPC@627207751))) after lookup
18:22:36.299 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)] connecting to peer
18:22:36.301 [DEBUG - kudu-nio-4] (Connection.java:757) [peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)] Successfully connected to peer
18:22:36.318 [DEBUG - kudu-nio-4] (Negotiator.java:918) Authenticated connection [id: 0x5b2f3ec1, L:/127.0.0.1:50034 - R:/127.18.238.129:41173] using TOKEN/null
18:22:36.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:36.327659 22002 raft_consensus.cc:1275] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 10 FOLLOWER]: Refusing update from remote peer 708d740e01414324ad249bf1854e9e5e: Log matching property violated. Preceding OpId in replica: term: 8 index: 32. Preceding OpId from leader: term: 10 index: 38. (index mismatch)
18:22:36.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:36.329061 21712 consensus_peers.cc:597] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e -> Peer 489d187f67e24039b9c5ea35f7d1e11e (127.18.238.131:46557): Couldn't send request to peer 489d187f67e24039b9c5ea35f7d1e11e. Status: Network error: Client connection negotiation failed: client connection to 127.18.238.131:46557: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
18:22:36.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:36.329237 22073 consensus_queue.cc:1048] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 32, Time since last communication: 0.000s
18:22:36.388 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.18.238.131:46557
18:22:36.388 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

18:22:36.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:36.389968 19386 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu
18:22:36.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu
18:22:36.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-2/wal
18:22:36.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-2/data
18:22:36.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
18:22:36.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
18:22:36.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
18:22:36.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:22:36.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:22:36.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:22:36.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-2/logs
18:22:36.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-2/data/info.pb
18:22:36.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:22:36.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
18:22:36.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
18:22:36.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
18:22:36.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
18:22:36.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:22:36.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:22:36.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:22:36.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
18:22:36.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
18:22:36.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.18.238.131:46557
18:22:36.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.18.238.131
18:22:36.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.18.238.190:40173,127.18.238.189:35025,127.18.238.188:40453
18:22:36.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=34759
18:22:36.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.18.238.131
18:22:36.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.18.238.148:35877
18:22:36.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:22:36.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:22:36.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
18:22:36.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:36.698717 22097 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
18:22:36.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:36.699357 22097 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
18:22:36.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:36.699991 22097 flags.cc:432] Enabled unsafe flag: --never_fsync=true
18:22:36.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:36.736596 22097 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
18:22:36.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:36.737468 22097 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.238.131
18:22:36.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:36.773259 22097 tablet_server_runner.cc:78] Tablet server non-default flags:
18:22:36.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-2/logs
18:22:36.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:22:36.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
18:22:36.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:22:36.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.18.238.148:35877
18:22:36.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:22:36.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
18:22:36.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-2/data
18:22:36.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-2/wal
18:22:36.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:22:36.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:22:36.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.18.238.131:46557
18:22:36.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
18:22:36.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:22:36.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:22:36.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-2/data/info.pb
18:22:36.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.18.238.131
18:22:36.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=34759
18:22:36.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.18.238.190:40173,127.18.238.189:35025,127.18.238.188:40453
18:22:36.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
18:22:36.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:22:36.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
18:22:36.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
18:22:36.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:22:36.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.18.238.131
18:22:36.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
18:22:36.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
18:22:36.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
18:22:36.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision cfdbd367ce7fd9747cb6f1860fc64263d9f9f79a
18:22:36.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
18:22:36.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 11 Aug 2026 18:01:34 UTC on bdcb31816ec0
18:22:36.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 17632
18:22:36.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
18:22:36.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:36.774955 22097 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
18:22:36.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:36.776554 22097 file_cache.cc:504] Constructed file cache file cache with capacity 419430
18:22:36.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:36.789656 22103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:22:37.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:36.792054 22104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:22:37.964 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:37.963876 22106 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
18:22:37.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:37.965409 22107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:22:37.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:37.965409 22097 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
18:22:37.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:37.967873 22097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
18:22:37.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:37.969966 22097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
18:22:37.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:37.971306 22097 hybrid_clock.cc:648] HybridClock initialized: now 1786472557971272 us; error 40 us; skew 500 ppm
18:22:37.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:37.972115 22097 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
18:22:37.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:37.977919 22097 webserver.cc:533] Webserver started at http://127.18.238.131:34759/ using document root <none> and password file <none>
18:22:37.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:37.978778 22097 fs_manager.cc:362] Metadata directory not provided
18:22:37.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:37.979014 22097 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
18:22:37.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:37.987131 22097 fs_manager.cc:714] Time spent opening directory manager: real 0.005s	user 0.004s	sys 0.000s
18:22:37.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:37.991564 22117 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
18:22:37.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:37.992808 22097 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.004s	sys 0.000s
18:22:37.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:37.993270 22097 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-2/data,/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-2/wal
18:22:37.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "489d187f67e24039b9c5ea35f7d1e11e"
18:22:37.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 18:21:52 on dist-test-slave-6nmv"
18:22:37.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:37.995318 22097 fs_report.cc:389] FS layout report
18:22:37.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
18:22:37.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-2/wal
18:22:37.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-2/wal
18:22:37.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-2/data/data
18:22:37.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
18:22:37.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
18:22:37.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
18:22:37.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
18:22:37.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
18:22:37.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
18:22:37.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
18:22:37.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
18:22:37.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
18:22:37.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
18:22:37.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
18:22:38.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:38.058460 22097 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
18:22:38.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:38.060029 22097 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
18:22:38.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:38.060519 22097 kserver.cc:163] Server-wide thread pool size limit: 3276
18:22:38.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:38.063647 22097 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
18:22:38.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:38.070243 22124 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
18:22:38.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:38.083078 22097 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
18:22:38.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:38.083348 22097 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.015s	user 0.000s	sys 0.002s
18:22:38.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:38.083613 22097 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
18:22:38.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:38.088389 22097 ts_tablet_manager.cc:616] Registered 1 tablets
18:22:38.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:38.088609 22097 ts_tablet_manager.cc:595] Time spent register tablets: real 0.005s	user 0.002s	sys 0.001s
18:22:38.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:38.089067 22124 tablet_bootstrap.cc:492] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e: Bootstrap starting.
18:22:38.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:38.170277 22124 log.cc:826] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e: Log is configured to *not* fsync() on all Append() calls
18:22:38.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:38.289930 22124 tablet_bootstrap.cc:492] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e: Bootstrap replayed 1/1 log segments. Stats: ops{read=36 overwritten=0 applied=36 ignored=0} inserts{seen=27 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
18:22:38.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:38.290998 22124 tablet_bootstrap.cc:492] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e: Bootstrap complete.
18:22:38.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:38.291113 22252 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.238.131:46557 every 8 connection(s)
18:22:38.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:38.290966 22097 rpc_server.cc:307] RPC server started. Bound to: 127.18.238.131:46557
18:22:38.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:38.293619 22124 ts_tablet_manager.cc:1403] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e: Time spent bootstrapping tablet: real 0.205s	user 0.153s	sys 0.044s
18:22:38.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:38.295506 22097 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-2/data/info.pb
18:22:38.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:38.297981 19386 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu as pid 22097
18:22:38.299 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
18:22:38.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:38.322297 22124 raft_consensus.cc:359] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 9 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:22:38.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:38.325554 22124 raft_consensus.cc:740] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 9 FOLLOWER]: Becoming Follower/Learner. State: Replica: 489d187f67e24039b9c5ea35f7d1e11e, State: Initialized, Role: FOLLOWER
18:22:38.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:38.326551 22124 consensus_queue.cc:260] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 36, Last appended: 9.36, Last appended by leader: 36, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:22:38.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:38.338176 22124 ts_tablet_manager.cc:1434] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e: Time spent starting tablet: real 0.044s	user 0.028s	sys 0.007s
18:22:38.394 [DEBUG - kudu-nio-4] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 30 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897, tablet=a6f8d542eda4472e9df5c193be9aa15d, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
18:22:38.395 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.18.238.129:41173
18:22:38.396 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

18:22:38.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:38.396847 19386 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu with pid 21680
18:22:38.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:38.406662 22255 heartbeater.cc:344] Connected to a master server at 127.18.238.190:40173
18:22:38.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:38.407188 22255 heartbeater.cc:461] Registering TS with master...
18:22:38.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:38.408618 22255 heartbeater.cc:507] Master 127.18.238.190:40173 requested a full tablet report, sending...
18:22:38.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:38.411002 22253 heartbeater.cc:344] Connected to a master server at 127.18.238.189:35025
18:22:38.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:38.411528 22253 heartbeater.cc:461] Registering TS with master...
18:22:38.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:38.412839 22253 heartbeater.cc:507] Master 127.18.238.189:35025 requested a full tablet report, sending...
18:22:38.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:38.416769 19524 ts_manager.cc:194] Re-registered known tserver with Master: 489d187f67e24039b9c5ea35f7d1e11e (127.18.238.131:46557)
18:22:38.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:38.417903 19440 ts_manager.cc:194] Re-registered known tserver with Master: 489d187f67e24039b9c5ea35f7d1e11e (127.18.238.131:46557)
18:22:38.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:38.419158 22254 heartbeater.cc:344] Connected to a master server at 127.18.238.188:40453
18:22:38.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:38.419533 22254 heartbeater.cc:461] Registering TS with master...
18:22:38.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:38.420508 22254 heartbeater.cc:507] Master 127.18.238.188:40453 requested a full tablet report, sending...
18:22:38.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:38.423333 19625 ts_manager.cc:194] Re-registered known tserver with Master: 489d187f67e24039b9c5ea35f7d1e11e (127.18.238.131:46557)
18:22:38.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:38.425601 19625 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.18.238.131:52511
18:22:38.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:38.428977 22254 heartbeater.cc:499] Master 127.18.238.188:40453 was elected leader, sending a full tablet report...
18:22:38.449 [DEBUG - kudu-nio-4] (Connection.java:239) [peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)] handling channelInactive
18:22:38.449 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)] cleaning up while in state READY due to: connection closed
18:22:38.450 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
18:22:38.450 [DEBUG - Time-limited test] (Connection.java:748) [peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)] connecting to peer
18:22:38.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:38.451128 22258 negotiation.cc:336] Failed RPC negotiation. Trace:
18:22:38.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 18:22:38.316003 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.18.238.129:34999 (local address 127.18.238.131:46557)
18:22:38.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 18:22:38.402691 (+ 86688us) server_negotiation.cc:207] Beginning negotiation
18:22:38.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 18:22:38.402724 (+    33us) server_negotiation.cc:400] Waiting for connection header
18:22:38.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 18:22:38.402780 (+    56us) server_negotiation.cc:408] Connection header received
18:22:38.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 18:22:38.405017 (+  2237us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
18:22:38.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 18:22:38.405082 (+    65us) server_negotiation.cc:462] Received NEGOTIATE request from client
18:22:38.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 18:22:38.405527 (+   445us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
18:22:38.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 18:22:38.405738 (+   211us) server_negotiation.cc:227] Negotiated authn=SASL
18:22:38.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 18:22:38.449249 (+ 43511us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.238.129:34999: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
18:22:38.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":70946,"thread_start_us":68774,"threads_started":1}
18:22:38.452 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:22:38.452 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)] cleaning up while in state CONNECTING due to: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:22:38.452 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173) for tablet a6f8d542eda4472e9df5c193be9aa15d: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:22:38.458 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:22:38.458 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737),489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557),708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)[L]]}]
18:22:38.458 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=a6f8d542eda4472e9df5c193be9aa15d, attempt=2, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=8), Traces: [0ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [2ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [2ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [4ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [8ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=Deferred@318502607(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007f5c7c36b7a8@24dda281 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@77088d03 -> wakeup thread Time-limited test)) after lookup
18:22:38.458 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)] connecting to peer
18:22:38.459 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:22:38.459 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)] cleaning up while in state CONNECTING due to: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:22:38.460 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173) for tablet a6f8d542eda4472e9df5c193be9aa15d: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:22:38.477 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:22:38.477 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737),489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557),708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)[L]]}]
18:22:38.477 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=a6f8d542eda4472e9df5c193be9aa15d, attempt=4, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=27), Traces: [0ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [2ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [2ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [4ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [8ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [8ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [9ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [10ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [27ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=Deferred@318502607(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007f5c7c36b7a8@24dda281 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@557815364 after retry RPC@317487093), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@77088d03 -> wakeup thread Time-limited test -> (continuation of Deferred@557815364 after retry RPC@317487093))) after lookup
18:22:38.477 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)] connecting to peer
18:22:38.478 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:22:38.478 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)] cleaning up while in state CONNECTING due to: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:22:38.478 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173) for tablet a6f8d542eda4472e9df5c193be9aa15d: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:22:38.518 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:22:38.518 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737),489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557),708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)[L]]}]
18:22:38.518 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=a6f8d542eda4472e9df5c193be9aa15d, attempt=6, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=68), Traces: [0ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [2ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [2ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [4ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [8ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [8ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [9ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [10ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [27ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [27ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [28ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [28ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [64ms] refreshing cache from master, [65ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [68ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=Deferred@318502607(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007f5c7c36b7a8@24dda281 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@557815364 after retry RPC@317487093) -> (continuation of Deferred@80254006 after retry RPC@1978275973), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@77088d03 -> wakeup thread Time-limited test -> (continuation of Deferred@557815364 after retry RPC@317487093) -> (continuation of Deferred@80254006 after retry RPC@1978275973))) after lookup
18:22:38.519 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)] connecting to peer
18:22:38.520 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:22:38.520 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)] cleaning up while in state CONNECTING due to: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:22:38.520 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173) for tablet a6f8d542eda4472e9df5c193be9aa15d: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:22:38.618 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:22:38.618 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737),489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557),708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)[L]]}]
18:22:38.619 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=a6f8d542eda4472e9df5c193be9aa15d, attempt=8, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=168), Traces: [0ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [2ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [2ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [4ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [8ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [8ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [9ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [10ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [27ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [27ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [28ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [28ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [64ms] refreshing cache from master, [65ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [68ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [69ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [70ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [70ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [164ms] refreshing cache from master, [165ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [168ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=Deferred@318502607(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007f5c7c36b7a8@24dda281 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@557815364 after retry RPC@317487093) -> (continuation of Deferred@80254006 after retry RPC@1978275973) -> (continuation of Deferred@2089673373 after retry RPC@1502016441), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@77088d03 -> wakeup thread Time-limited test -> (continuation of Deferred@557815364 after retry RPC@317487093) -> (continuation of Deferred@80254006 after retry RPC@1978275973) -> (continuation of Deferred@2089673373 after retry RPC@1502016441))) after lookup
18:22:38.619 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)] connecting to peer
18:22:38.620 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:22:38.620 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)] cleaning up while in state CONNECTING due to: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:22:38.620 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173) for tablet a6f8d542eda4472e9df5c193be9aa15d: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:22:38.937 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:22:38.938 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737),489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557),708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)[L]]}]
18:22:38.938 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=a6f8d542eda4472e9df5c193be9aa15d, attempt=10, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=488), Traces: [0ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [2ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [2ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [4ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [8ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [8ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [9ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [10ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [27ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [27ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [28ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [28ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [64ms] refreshing cache from master, [65ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [68ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [69ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [70ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [70ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [164ms] refreshing cache from master, [165ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [168ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [169ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [170ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [170ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [484ms] refreshing cache from master, [484ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [487ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=Deferred@318502607(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007f5c7c36b7a8@24dda281 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@557815364 after retry RPC@317487093) -> (continuation of Deferred@80254006 after retry RPC@1978275973) -> (continuation of Deferred@2089673373 after retry RPC@1502016441) -> (continuation of Deferred@1268064053 after retry RPC@635446981), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@77088d03 -> wakeup thread Time-limited test -> (continuation of Deferred@557815364 after retry RPC@317487093) -> (continuation of Deferred@80254006 after retry RPC@1978275973) -> (continuation of Deferred@2089673373 after retry RPC@1502016441) -> (continuation of Deferred@1268064053 after retry RPC@635446981))) after lookup
18:22:38.939 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)] connecting to peer
18:22:38.940 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:22:38.940 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)] cleaning up while in state CONNECTING due to: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:22:38.940 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173) for tablet a6f8d542eda4472e9df5c193be9aa15d: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173
18:22:39.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:39.535502 22268 raft_consensus.cc:493] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
18:22:39.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:39.535902 22268 raft_consensus.cc:515] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:22:39.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:39.538208 22268 leader_election.cc:290] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 24001d6c96764c04999eb5de15b9eba9 (127.18.238.130:35737), 708d740e01414324ad249bf1854e9e5e (127.18.238.129:41173)
18:22:39.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:39.542927 22120 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.238.129:41173: connect: Connection refused (error 111)
18:22:39.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:39.546518 22120 leader_election.cc:336] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 708d740e01414324ad249bf1854e9e5e (127.18.238.129:41173): Network error: Client connection negotiation failed: client connection to 127.18.238.129:41173: connect: Connection refused (error 111)
18:22:39.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:39.549795 22002 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6f8d542eda4472e9df5c193be9aa15d" candidate_uuid: "489d187f67e24039b9c5ea35f7d1e11e" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "24001d6c96764c04999eb5de15b9eba9" is_pre_election: true
18:22:39.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:39.551087 22120 leader_election.cc:304] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 489d187f67e24039b9c5ea35f7d1e11e; no voters: 24001d6c96764c04999eb5de15b9eba9, 708d740e01414324ad249bf1854e9e5e
18:22:39.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:39.551766 22268 raft_consensus.cc:2749] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
18:22:39.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:39.760324 22273 raft_consensus.cc:493] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 708d740e01414324ad249bf1854e9e5e)
18:22:39.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:39.760754 22273 raft_consensus.cc:515] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:22:39.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:39.763404 22273 leader_election.cc:290] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 489d187f67e24039b9c5ea35f7d1e11e (127.18.238.131:46557), 708d740e01414324ad249bf1854e9e5e (127.18.238.129:41173)
18:22:39.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:39.767267 21927 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.238.129:41173: connect: Connection refused (error 111)
18:22:39.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:39.771157 21927 leader_election.cc:336] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 708d740e01414324ad249bf1854e9e5e (127.18.238.129:41173): Network error: Client connection negotiation failed: client connection to 127.18.238.129:41173: connect: Connection refused (error 111)
18:22:39.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:39.775602 22199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6f8d542eda4472e9df5c193be9aa15d" candidate_uuid: "24001d6c96764c04999eb5de15b9eba9" candidate_term: 11 candidate_status { last_received { term: 10 index: 40 } } ignore_live_leader: false dest_uuid: "489d187f67e24039b9c5ea35f7d1e11e" is_pre_election: true
18:22:39.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:39.776223 22199 raft_consensus.cc:2468] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24001d6c96764c04999eb5de15b9eba9 in term 9.
18:22:39.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:39.777437 21924 leader_election.cc:304] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24001d6c96764c04999eb5de15b9eba9, 489d187f67e24039b9c5ea35f7d1e11e; no voters: 708d740e01414324ad249bf1854e9e5e
18:22:39.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:39.778112 22273 raft_consensus.cc:2804] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 10 FOLLOWER]: Leader pre-election won for term 11
18:22:39.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:39.778434 22273 raft_consensus.cc:493] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 708d740e01414324ad249bf1854e9e5e)
18:22:39.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:39.778725 22273 raft_consensus.cc:3060] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 10 FOLLOWER]: Advancing to term 11
18:22:39.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:39.783329 22273 raft_consensus.cc:515] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:22:39.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:39.784752 22273 leader_election.cc:290] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [CANDIDATE]: Term 11 election: Requested vote from peers 489d187f67e24039b9c5ea35f7d1e11e (127.18.238.131:46557), 708d740e01414324ad249bf1854e9e5e (127.18.238.129:41173)
18:22:39.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:39.785293 22199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6f8d542eda4472e9df5c193be9aa15d" candidate_uuid: "24001d6c96764c04999eb5de15b9eba9" candidate_term: 11 candidate_status { last_received { term: 10 index: 40 } } ignore_live_leader: false dest_uuid: "489d187f67e24039b9c5ea35f7d1e11e"
18:22:39.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:39.785615 22199 raft_consensus.cc:3060] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 9 FOLLOWER]: Advancing to term 11
18:22:39.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:39.789063 21927 leader_election.cc:336] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 708d740e01414324ad249bf1854e9e5e (127.18.238.129:41173): Network error: Client connection negotiation failed: client connection to 127.18.238.129:41173: connect: Connection refused (error 111)
18:22:39.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:39.797010 22199 raft_consensus.cc:2468] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24001d6c96764c04999eb5de15b9eba9 in term 11.
18:22:39.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:39.798127 21924 leader_election.cc:304] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24001d6c96764c04999eb5de15b9eba9, 489d187f67e24039b9c5ea35f7d1e11e; no voters: 708d740e01414324ad249bf1854e9e5e
18:22:39.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:39.798759 22273 raft_consensus.cc:2804] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 11 FOLLOWER]: Leader election won for term 11
18:22:39.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:39.800436 22273 raft_consensus.cc:697] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [term 11 LEADER]: Becoming Leader. State: Replica: 24001d6c96764c04999eb5de15b9eba9, State: Running, Role: LEADER
18:22:39.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:39.801349 22273 consensus_queue.cc:237] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 10.40, Last appended by leader: 40, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:22:39.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:39.811112 19625 catalog_manager.cc:5719] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 reported cstate change: term changed from 10 to 11, leader changed from 708d740e01414324ad249bf1854e9e5e (127.18.238.129) to 24001d6c96764c04999eb5de15b9eba9 (127.18.238.130). New cstate: current_term: 11 leader_uuid: "24001d6c96764c04999eb5de15b9eba9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } health_report { overall_health: UNKNOWN } } }
18:22:40.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:40.294523 21927 consensus_peers.cc:597] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 -> Peer 708d740e01414324ad249bf1854e9e5e (127.18.238.129:41173): Couldn't send request to peer 708d740e01414324ad249bf1854e9e5e. Status: Network error: Client connection negotiation failed: client connection to 127.18.238.129:41173: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
18:22:40.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:40.359702 22199 raft_consensus.cc:1275] T a6f8d542eda4472e9df5c193be9aa15d P 489d187f67e24039b9c5ea35f7d1e11e [term 11 FOLLOWER]: Refusing update from remote peer 24001d6c96764c04999eb5de15b9eba9: Log matching property violated. Preceding OpId in replica: term: 9 index: 36. Preceding OpId from leader: term: 11 index: 41. (index mismatch)
18:22:40.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:40.361001 22273 consensus_queue.cc:1048] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 36, Time since last communication: 0.000s
18:22:40.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:40.372464 22295 mvcc.cc:204] Tried to move back new op lower bound from 7317391590701559808 to 7317391590217641984. Current Snapshot: MvccSnapshot[applied={T|T < 7317391590701559808}]
18:22:40.958 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet a6f8d542eda4472e9df5c193be9aa15d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897' with partition [<start>, <end>)
18:22:40.958 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=a6f8d542eda4472e9df5c193be9aa15d@[24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)[L],489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557),708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)]}]
18:22:40.959 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=a6f8d542eda4472e9df5c193be9aa15d, attempt=12, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=2508), Traces: [0ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [2ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [2ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [4ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [8ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [8ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [9ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [10ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [27ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [27ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [28ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [28ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [64ms] refreshing cache from master, [65ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [68ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [69ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [70ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [70ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [164ms] refreshing cache from master, [165ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [168ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [169ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [170ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [170ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [484ms] refreshing cache from master, [484ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [487ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, [489ms] sending RPC to server 708d740e01414324ad249bf1854e9e5e, [490ms] received response from server 708d740e01414324ad249bf1854e9e5e: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [490ms] delaying RPC due to: Network error: Failed to connect to peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173): Connection refused: /127.18.238.129:41173, [2504ms] refreshing cache from master, [2504ms] Sub RPC GetTableLocations: sending RPC to server master-127.18.238.188:40453, [2508ms] Sub RPC GetTableLocations: received response from server master-127.18.238.188:40453: OK, deferred=Deferred@318502607(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007f5c7c36b7a8@24dda281 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@557815364 after retry RPC@317487093) -> (continuation of Deferred@80254006 after retry RPC@1978275973) -> (continuation of Deferred@2089673373 after retry RPC@1502016441) -> (continuation of Deferred@1268064053 after retry RPC@635446981) -> (continuation of Deferred@113976301 after retry RPC@49892064), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@77088d03 -> wakeup thread Time-limited test -> (continuation of Deferred@557815364 after retry RPC@317487093) -> (continuation of Deferred@80254006 after retry RPC@1978275973) -> (continuation of Deferred@2089673373 after retry RPC@1502016441) -> (continuation of Deferred@1268064053 after retry RPC@635446981) -> (continuation of Deferred@113976301 after retry RPC@49892064))) after lookup
18:22:40.959 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)] connecting to peer
18:22:40.960 [DEBUG - kudu-nio-3] (Connection.java:757) [peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)] Successfully connected to peer
18:22:40.979 [DEBUG - kudu-nio-3] (Negotiator.java:918) Authenticated connection [id: 0x2df47619, L:/127.0.0.1:51052 - R:/127.18.238.130:35737] using TOKEN/null
18:22:41.019 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.18.238.129:41173
18:22:41.020 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

18:22:41.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:41.021342 19386 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu
18:22:41.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu
18:22:41.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/wal
18:22:41.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/data
18:22:41.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
18:22:41.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
18:22:41.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
18:22:41.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:22:41.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:22:41.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:22:41.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/logs
18:22:41.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/data/info.pb
18:22:41.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:22:41.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
18:22:41.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
18:22:41.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
18:22:41.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
18:22:41.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:22:41.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:22:41.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:22:41.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
18:22:41.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
18:22:41.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.18.238.129:41173
18:22:41.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.18.238.129
18:22:41.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.18.238.190:40173,127.18.238.189:35025,127.18.238.188:40453
18:22:41.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44313
18:22:41.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.18.238.129
18:22:41.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.18.238.148:35877
18:22:41.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:22:41.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:22:41.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
18:22:41.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:41.339059 22314 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
18:22:41.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:41.339466 22314 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
18:22:41.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:41.339849 22314 flags.cc:432] Enabled unsafe flag: --never_fsync=true
18:22:41.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:41.369838 22314 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
18:22:41.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:41.370814 22314 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.238.129
18:22:41.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:41.410954 22314 tablet_server_runner.cc:78] Tablet server non-default flags:
18:22:41.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/logs
18:22:41.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:22:41.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
18:22:41.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:22:41.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.18.238.148:35877
18:22:41.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:22:41.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
18:22:41.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/data
18:22:41.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/wal
18:22:41.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:22:41.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:22:41.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.18.238.129:41173
18:22:41.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
18:22:41.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:22:41.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:22:41.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/data/info.pb
18:22:41.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.18.238.129
18:22:41.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44313
18:22:41.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.18.238.190:40173,127.18.238.189:35025,127.18.238.188:40453
18:22:41.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
18:22:41.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:22:41.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
18:22:41.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
18:22:41.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:22:41.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.18.238.129
18:22:41.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
18:22:41.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
18:22:41.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
18:22:41.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision cfdbd367ce7fd9747cb6f1860fc64263d9f9f79a
18:22:41.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
18:22:41.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 11 Aug 2026 18:01:34 UTC on bdcb31816ec0
18:22:41.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 17632
18:22:41.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
18:22:41.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:41.412314 22314 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
18:22:41.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:41.413946 22314 file_cache.cc:504] Constructed file cache file cache with capacity 419430
18:22:41.428 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:41.427887 22324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:22:42.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:41.428444 22325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:22:42.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:42.631896 22327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:22:42.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:42.634380 22326 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1201 milliseconds
18:22:42.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:42.634456 22314 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
18:22:42.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:42.635735 22314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
18:22:42.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:42.638211 22314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
18:22:42.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:42.639577 22314 hybrid_clock.cc:648] HybridClock initialized: now 1786472562639521 us; error 31 us; skew 500 ppm
18:22:42.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:42.640412 22314 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
18:22:42.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:42.646341 22314 webserver.cc:533] Webserver started at http://127.18.238.129:44313/ using document root <none> and password file <none>
18:22:42.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:42.647141 22314 fs_manager.cc:362] Metadata directory not provided
18:22:42.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:42.647339 22314 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
18:22:42.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:42.659402 22314 fs_manager.cc:714] Time spent opening directory manager: real 0.007s	user 0.007s	sys 0.001s
18:22:42.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:42.659437 21927 consensus_peers.cc:597] T a6f8d542eda4472e9df5c193be9aa15d P 24001d6c96764c04999eb5de15b9eba9 -> Peer 708d740e01414324ad249bf1854e9e5e (127.18.238.129:41173): Couldn't send request to peer 708d740e01414324ad249bf1854e9e5e. Status: Network error: Client connection negotiation failed: client connection to 127.18.238.129:41173: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
18:22:42.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:42.665484 22337 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
18:22:42.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:42.667225 22314 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.003s	sys 0.000s
18:22:42.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:42.667744 22314 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/data,/tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/wal
18:22:42.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "708d740e01414324ad249bf1854e9e5e"
18:22:42.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 18:21:47 on dist-test-slave-6nmv"
18:22:42.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:42.670414 22314 fs_report.cc:389] FS layout report
18:22:42.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
18:22:42.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/wal
18:22:42.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/wal
18:22:42.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/data/data
18:22:42.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
18:22:42.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
18:22:42.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
18:22:42.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
18:22:42.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
18:22:42.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
18:22:42.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
18:22:42.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
18:22:42.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
18:22:42.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
18:22:42.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
18:22:42.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:42.751701 22314 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
18:22:42.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:42.753433 22314 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
18:22:42.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:42.754027 22314 kserver.cc:163] Server-wide thread pool size limit: 3276
18:22:42.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:42.756476 22314 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
18:22:42.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:42.761984 22348 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
18:22:42.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:42.769668 22314 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
18:22:42.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:42.769907 22314 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.009s	user 0.002s	sys 0.000s
18:22:42.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:42.770243 22314 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
18:22:42.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:42.774899 22314 ts_tablet_manager.cc:616] Registered 1 tablets
18:22:42.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:42.775094 22314 ts_tablet_manager.cc:595] Time spent register tablets: real 0.005s	user 0.001s	sys 0.003s
18:22:42.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:42.775449 22348 tablet_bootstrap.cc:492] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e: Bootstrap starting.
18:22:42.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:42.844964 22348 log.cc:826] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e: Log is configured to *not* fsync() on all Append() calls
18:22:42.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:42.980381 22314 rpc_server.cc:307] RPC server started. Bound to: 127.18.238.129:41173
18:22:42.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:42.980485 22475 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.238.129:41173 every 8 connection(s)
18:22:42.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:42.981174 22348 tablet_bootstrap.cc:492] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e: Bootstrap replayed 1/1 log segments. Stats: ops{read=40 overwritten=0 applied=40 ignored=0} inserts{seen=30 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
18:22:42.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:42.981971 22348 tablet_bootstrap.cc:492] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e: Bootstrap complete.
18:22:42.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:42.983172 22314 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskpuxqmP/test-tmp/mini-kudu-cluster564963583466424744/ts-0/data/info.pb
18:22:42.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:42.983736 22348 ts_tablet_manager.cc:1403] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e: Time spent bootstrapping tablet: real 0.209s	user 0.149s	sys 0.051s
18:22:42.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:42.990631 19386 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu as pid 22314
18:22:42.992 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
18:22:43.018 [DEBUG - kudu-nio-3] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 33 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897, tablet=a6f8d542eda4472e9df5c193be9aa15d, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
18:22:43.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:43.015774 22348 raft_consensus.cc:359] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [term 10 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:22:43.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:43.020912 22348 raft_consensus.cc:740] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [term 10 FOLLOWER]: Becoming Follower/Learner. State: Replica: 708d740e01414324ad249bf1854e9e5e, State: Initialized, Role: FOLLOWER
18:22:43.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:43.024080 22348 consensus_queue.cc:260] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 40, Last appended: 10.40, Last appended by leader: 40, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24001d6c96764c04999eb5de15b9eba9" member_type: VOTER last_known_addr { host: "127.18.238.130" port: 35737 } } peers { permanent_uuid: "489d187f67e24039b9c5ea35f7d1e11e" member_type: VOTER last_known_addr { host: "127.18.238.131" port: 46557 } } peers { permanent_uuid: "708d740e01414324ad249bf1854e9e5e" member_type: VOTER last_known_addr { host: "127.18.238.129" port: 41173 } }
18:22:43.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:43.030879 22476 heartbeater.cc:344] Connected to a master server at 127.18.238.189:35025
18:22:43.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:43.031423 22476 heartbeater.cc:461] Registering TS with master...
18:22:43.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:43.032996 22476 heartbeater.cc:507] Master 127.18.238.189:35025 requested a full tablet report, sending...
18:22:43.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:43.038514 22477 heartbeater.cc:344] Connected to a master server at 127.18.238.188:40453
18:22:43.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:43.038836 22477 heartbeater.cc:461] Registering TS with master...
18:22:43.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:43.039706 22477 heartbeater.cc:507] Master 127.18.238.188:40453 requested a full tablet report, sending...
18:22:43.042 [DEBUG - kudu-nio-3] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 33 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786472512897, tablet=a6f8d542eda4472e9df5c193be9aa15d, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
18:22:43.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:43.040607 22478 heartbeater.cc:344] Connected to a master server at 127.18.238.190:40173
18:22:43.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:43.040970 22478 heartbeater.cc:461] Registering TS with master...
18:22:43.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:43.041803 22478 heartbeater.cc:507] Master 127.18.238.190:40173 requested a full tablet report, sending...
18:22:43.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:43.042935 22348 ts_tablet_manager.cc:1434] T a6f8d542eda4472e9df5c193be9aa15d P 708d740e01414324ad249bf1854e9e5e: Time spent starting tablet: real 0.059s	user 0.020s	sys 0.016s
18:22:43.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:43.045746 19440 ts_manager.cc:194] Re-registered known tserver with Master: 708d740e01414324ad249bf1854e9e5e (127.18.238.129:41173)
18:22:43.049 [DEBUG - main] (Connection.java:575) [peer master-127.18.238.188:40453(127.18.238.188:40453)] disconnecting while in state READY
18:22:43.050 [DEBUG - main] (Connection.java:575) [peer 489d187f67e24039b9c5ea35f7d1e11e(127.18.238.131:46557)] disconnecting while in state TERMINATED
18:22:43.050 [DEBUG - main] (Connection.java:575) [peer 708d740e01414324ad249bf1854e9e5e(127.18.238.129:41173)] disconnecting while in state TERMINATED
18:22:43.050 [DEBUG - main] (Connection.java:575) [peer master-127.18.238.190:40173(127.18.238.190:40173)] disconnecting while in state READY
18:22:43.050 [DEBUG - main] (Connection.java:575) [peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)] disconnecting while in state READY
18:22:43.050 [DEBUG - main] (Connection.java:575) [peer master-127.18.238.189:35025(127.18.238.189:35025)] disconnecting while in state READY
18:22:43.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:43.049998 19526 ts_manager.cc:194] Re-registered known tserver with Master: 708d740e01414324ad249bf1854e9e5e (127.18.238.129:41173)
18:22:43.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:43.049968 19625 ts_manager.cc:194] Re-registered known tserver with Master: 708d740e01414324ad249bf1854e9e5e (127.18.238.129:41173)
18:22:43.052 [DEBUG - kudu-nio-2] (Connection.java:239) [peer master-127.18.238.188:40453(127.18.238.188:40453)] handling channelInactive
18:22:43.053 [DEBUG - kudu-nio-3] (Connection.java:239) [peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)] handling channelInactive
18:22:43.053 [DEBUG - kudu-nio-2] (Connection.java:709) [peer master-127.18.238.188:40453(127.18.238.188:40453)] cleaning up while in state READY due to: connection closed
18:22:43.053 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 24001d6c96764c04999eb5de15b9eba9(127.18.238.130:35737)] cleaning up while in state READY due to: connection closed
18:22:43.054 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.18.238.189:35025(127.18.238.189:35025)] handling channelInactive
18:22:43.054 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.18.238.189:35025(127.18.238.189:35025)] cleaning up while in state READY due to: connection closed
18:22:43.055 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2704) Releasing all remaining resources
18:22:43.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:43.056137 19625 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.18.238.129:35917
18:22:43.057 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.18.238.190:40173(127.18.238.190:40173)] handling channelInactive
18:22:43.057 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.18.238.190:40173(127.18.238.190:40173)] cleaning up while in state READY due to: connection closed
18:22:43.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:43.063155 22477 heartbeater.cc:499] Master 127.18.238.188:40453 was elected leader, sending a full tablet report...
18:22:43.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:43.065774 19386 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu with pid 22314
18:22:43.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:43.106456 19611 connection.cc:573] server connection from 127.18.238.129:35917 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
18:22:43.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:43.107717 19386 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu with pid 21901
18:22:43.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:22:43.107462 19509 connection.cc:573] server connection from 127.18.238.129:48067 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
18:22:43.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:43.137075 19386 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu with pid 22097
18:22:43.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:43.173326 19386 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu with pid 19402
18:22:43.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:43.211870 19386 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu with pid 19482
18:22:43.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:22:43.249393 19386 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskpuxqmP/build/tsan/bin/kudu with pid 19569
18:22:44.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-08-11T18:22:44Z chronyd exiting

Time: 68.245

OK (1 test)

