JUnit version 4.13.2
.07:56:21.253 [DEBUG - main] (RandomUtils.java:49) Using random seed: 1783497381246
07:56:21.306 [INFO - main] (RetryRule.java:138) Creating RetryStatement with result reporter and retry count of 0 (no retries)
07:56:21.362 [INFO - main] (KuduTestHarness.java:155) Creating a new MiniKuduCluster...
07:56:21.374 [INFO - main] (TempDirUtils.java:71) Using the temp directory defined by TEST_TMPDIR: /tmp/dist-test-task2rU0iV/test-tmp
07:56:21.380 [INFO - main] (KuduBinaryLocator.java:83) Using Kudu binary directory specified by system property 'kuduBinDir': ../../../build/asan/bin
07:56:21.383 [INFO - main] (MiniKuduCluster.java:217) Starting process: [/tmp/dist-test-task2rU0iV/java/build/dist-test/../../../build/asan/bin/kudu, test, mini_cluster, --serialization=pb]
07:56:21.659 [DEBUG - main] (MiniKuduCluster.java:185) Request: create_cluster {
  num_masters: 3
  num_tservers: 3
  enable_kerberos: false
  cluster_root: "/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629"
  hms_mode: NONE
  mini_kdc_options {
  }
  principal: "kudu"
  mini_oidc_options {
  }
}

07:56:21.849 [DEBUG - main] (MiniKuduCluster.java:195) Response: 
07:56:21.851 [DEBUG - main] (MiniKuduCluster.java:185) Request: start_cluster {
}

07:56:21.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:21.854950 26541 mini_oidc.cc:150] Starting JWKS server
07:56:21.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:21.858263 26541 webserver.cc:533] Webserver started at http://127.0.0.1:39705/ using document root <none> and password file <none>
07:56:21.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:21.858563 26541 mini_oidc.cc:186] Starting OIDC Discovery server
07:56:21.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:21.861227 26541 webserver.cc:533] Webserver started at http://127.0.0.1:36031/ using document root <none> and password file <none>
07:56:21.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-07-08T07:56:21Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
07:56:21.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-07-08T07:56:21Z Disabled control of system clock
07:56:21.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:21.893793 26541 external_mini_cluster.cc:1372] Running /tmp/dist-test-task2rU0iV/build/asan/bin/kudu
07:56:21.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task2rU0iV/build/asan/bin/kudu
07:56:21.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/master-0/wal
07:56:21.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/master-0/data
07:56:21.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
07:56:21.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
07:56:21.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
07:56:21.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:56:21.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:56:21.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:56:21.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/master-0/logs
07:56:21.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/master-0/data/info.pb
07:56:21.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:56:21.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
07:56:21.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
07:56:21.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
07:56:21.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
07:56:21.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:56:21.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:56:21.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:56:21.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
07:56:21.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
07:56:21.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
07:56:21.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
07:56:21.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.235.126:46873
07:56:21.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.235.126
07:56:21.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
07:56:21.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.235.84:44497
07:56:21.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:56:21.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:56:21.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
07:56:21.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
07:56:21.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.25.235.126:46873,127.25.235.125:42159,127.25.235.124:39273
07:56:21.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
07:56:21.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:39705/jwks.json with env {}
07:56:22.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:22.229212 26552 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
07:56:22.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:22.229529 26552 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
07:56:22.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:22.229607 26552 flags.cc:432] Enabled unsafe flag: --never_fsync=true
07:56:22.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:22.238395 26552 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
07:56:22.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:22.238497 26552 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
07:56:22.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:22.238550 26552 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
07:56:22.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:22.238634 26552 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
07:56:22.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:22.238718 26552 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
07:56:22.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:22.238766 26552 flags.cc:432] Enabled experimental flag: --jwks_url=http://localhost:39705/jwks.json
07:56:22.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.249816 26552 master_runner.cc:387] Master server non-default flags:
07:56:22.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:56:22.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.235.84:44497
07:56:22.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:56:22.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
07:56:22.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/master-0/data
07:56:22.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/master-0/wal
07:56:22.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
07:56:22.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.25.235.126:46873,127.25.235.125:42159,127.25.235.124:39273
07:56:22.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:56:22.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:56:22.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
07:56:22.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.235.126:46873
07:56:22.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
07:56:22.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
07:56:22.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
07:56:22.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:39705/jwks.json
07:56:22.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:56:22.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:56:22.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/master-0/data/info.pb
07:56:22.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.235.126
07:56:22.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
07:56:22.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
07:56:22.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:56:22.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
07:56:22.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
07:56:22.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:56:22.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/master-0/logs
07:56:22.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:56:22.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
07:56:22.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
07:56:22.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
07:56:22.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
07:56:22.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
07:56:22.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
07:56:22.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 07:43:22 UTC on bdcb31816ec0
07:56:22.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12770
07:56:22.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
07:56:22.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.251880 26552 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
07:56:22.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.254073 26552 file_cache.cc:504] Constructed file cache file cache with capacity 419430
07:56:22.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:22.264194 26557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:22.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:22.264182 26558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:22.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.268249 26552 server_base.cc:1061] running on GCE node
07:56:22.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:22.269059 26560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:22.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.270035 26552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
07:56:22.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.271723 26552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
07:56:22.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.272986 26552 hybrid_clock.cc:648] HybridClock initialized: now 1783497382272921 us; error 68 us; skew 500 ppm
07:56:22.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.273330 26552 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
07:56:22.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.275552 26552 webserver.cc:533] Webserver started at http://127.25.235.126:37817/ using document root <none> and password file <none>
07:56:22.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.276270 26552 fs_manager.cc:362] Metadata directory not provided
07:56:22.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.276358 26552 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
07:56:22.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.276731 26552 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
07:56:22.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.279276 26552 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/master-0/data/instance:
07:56:22.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "8b30b8ecb407419c96af767f8052e5ce"
07:56:22.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 07:56:22 on dist-test-slave-63m1"
07:56:22.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.280131 26552 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/master-0/wal/instance:
07:56:22.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "8b30b8ecb407419c96af767f8052e5ce"
07:56:22.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 07:56:22 on dist-test-slave-63m1"
07:56:22.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.285449 26552 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.003s	sys 0.005s
07:56:22.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.289012 26566 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
07:56:22.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.290957 26552 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.004s	sys 0.000s
07:56:22.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.291153 26552 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/master-0/data,/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/master-0/wal
07:56:22.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "8b30b8ecb407419c96af767f8052e5ce"
07:56:22.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 07:56:22 on dist-test-slave-63m1"
07:56:22.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.291458 26552 fs_report.cc:389] FS layout report
07:56:22.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
07:56:22.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/master-0/wal
07:56:22.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/master-0/wal
07:56:22.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/master-0/data/data
07:56:22.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
07:56:22.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
07:56:22.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
07:56:22.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
07:56:22.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
07:56:22.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
07:56:22.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
07:56:22.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
07:56:22.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
07:56:22.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
07:56:22.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
07:56:22.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.323181 26552 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
07:56:22.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.324183 26552 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
07:56:22.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.324509 26552 kserver.cc:163] Server-wide thread pool size limit: 3276
07:56:22.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.347615 26552 rpc_server.cc:307] RPC server started. Bound to: 127.25.235.126:46873
07:56:22.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.347621 26617 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.235.126:46873 every 8 connection(s)
07:56:22.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.349588 26552 server_base.cc:1193] Dumped server information to /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/master-0/data/info.pb
07:56:22.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.351485 26541 external_mini_cluster.cc:1434] Started /tmp/dist-test-task2rU0iV/build/asan/bin/kudu as pid 26552
07:56:22.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.351795 26541 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/master-0/wal/instance
07:56:22.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.354279 26618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
07:56:22.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.354295 26541 external_mini_cluster.cc:1372] Running /tmp/dist-test-task2rU0iV/build/asan/bin/kudu
07:56:22.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task2rU0iV/build/asan/bin/kudu
07:56:22.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/master-1/wal
07:56:22.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/master-1/data
07:56:22.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
07:56:22.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
07:56:22.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
07:56:22.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:56:22.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:56:22.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:56:22.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/master-1/logs
07:56:22.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/master-1/data/info.pb
07:56:22.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:56:22.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
07:56:22.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
07:56:22.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
07:56:22.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
07:56:22.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:56:22.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:56:22.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:56:22.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
07:56:22.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
07:56:22.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
07:56:22.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
07:56:22.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.235.125:42159
07:56:22.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.235.125
07:56:22.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
07:56:22.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.235.84:44497
07:56:22.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:56:22.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:56:22.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
07:56:22.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
07:56:22.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.25.235.126:46873,127.25.235.125:42159,127.25.235.124:39273
07:56:22.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
07:56:22.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:39705/jwks.json with env {}
07:56:22.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.361995 26618 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.235.126" port: 46873 } has no permanent_uuid. Determining permanent_uuid...
07:56:22.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.385812 26618 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.235.125" port: 42159 } has no permanent_uuid. Determining permanent_uuid...
07:56:22.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:22.387212 26570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.235.125:42159: connect: Connection refused (error 111)
07:56:22.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:22.388954 26618 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.235.125:42159: Network error: Client connection negotiation failed: client connection to 127.25.235.125:42159: connect: Connection refused (error 111)
07:56:22.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.438318 26618 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.235.125" port: 42159 } attempt: 1
07:56:22.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:22.440418 26618 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.235.125:42159: Network error: Client connection negotiation failed: client connection to 127.25.235.125:42159: connect: Connection refused (error 111)
07:56:22.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.508721 26618 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.235.125" port: 42159 } attempt: 2
07:56:22.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:22.511219 26618 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.235.125:42159: Network error: Client connection negotiation failed: client connection to 127.25.235.125:42159: connect: Connection refused (error 111)
07:56:22.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.602476 26618 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.235.125" port: 42159 } attempt: 3
07:56:22.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:22.604801 26618 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.235.125:42159: Network error: Client connection negotiation failed: client connection to 127.25.235.125:42159: connect: Connection refused (error 111)
07:56:22.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:22.700062 26620 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
07:56:22.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:22.700435 26620 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
07:56:22.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:22.700551 26620 flags.cc:432] Enabled unsafe flag: --never_fsync=true
07:56:22.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:22.709281 26620 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
07:56:22.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:22.709420 26620 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
07:56:22.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:22.709501 26620 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
07:56:22.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:22.709556 26620 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
07:56:22.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:22.709630 26620 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
07:56:22.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:22.709694 26620 flags.cc:432] Enabled experimental flag: --jwks_url=http://localhost:39705/jwks.json
07:56:22.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.720255 26620 master_runner.cc:387] Master server non-default flags:
07:56:22.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:56:22.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.235.84:44497
07:56:22.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:56:22.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
07:56:22.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/master-1/data
07:56:22.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/master-1/wal
07:56:22.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
07:56:22.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.25.235.126:46873,127.25.235.125:42159,127.25.235.124:39273
07:56:22.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:56:22.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:56:22.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
07:56:22.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.235.125:42159
07:56:22.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
07:56:22.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
07:56:22.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
07:56:22.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:39705/jwks.json
07:56:22.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:56:22.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:56:22.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/master-1/data/info.pb
07:56:22.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.235.125
07:56:22.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
07:56:22.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
07:56:22.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:56:22.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
07:56:22.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
07:56:22.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:56:22.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/master-1/logs
07:56:22.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:56:22.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
07:56:22.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
07:56:22.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
07:56:22.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
07:56:22.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
07:56:22.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
07:56:22.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 07:43:22 UTC on bdcb31816ec0
07:56:22.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12770
07:56:22.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
07:56:22.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.722622 26620 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
07:56:22.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.725397 26620 file_cache.cc:504] Constructed file cache file cache with capacity 419430
07:56:22.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:22.735052 26629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:22.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:22.736076 26628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:22.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.736725 26620 server_base.cc:1061] running on GCE node
07:56:22.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:22.736929 26631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:22.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.737927 26620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
07:56:22.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.740971 26620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
07:56:22.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.742246 26620 hybrid_clock.cc:648] HybridClock initialized: now 1783497382742174 us; error 71 us; skew 500 ppm
07:56:22.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.742756 26620 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
07:56:22.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.745914 26620 webserver.cc:533] Webserver started at http://127.25.235.125:36231/ using document root <none> and password file <none>
07:56:22.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.747012 26620 fs_manager.cc:362] Metadata directory not provided
07:56:22.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.747159 26620 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
07:56:22.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.747622 26620 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
07:56:22.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.748091 26618 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.235.125" port: 42159 } attempt: 4
07:56:22.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:22.750416 26618 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.235.125:42159: Network error: Client connection negotiation failed: client connection to 127.25.235.125:42159: connect: Connection refused (error 111)
07:56:22.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.751384 26620 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/master-1/data/instance:
07:56:22.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "947b6ed1d6bb49e49d50a03e028cbd24"
07:56:22.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 07:56:22 on dist-test-slave-63m1"
07:56:22.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.752607 26620 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/master-1/wal/instance:
07:56:22.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "947b6ed1d6bb49e49d50a03e028cbd24"
07:56:22.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 07:56:22 on dist-test-slave-63m1"
07:56:22.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.760675 26620 fs_manager.cc:696] Time spent creating directory manager: real 0.008s	user 0.005s	sys 0.001s
07:56:22.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.764983 26637 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
07:56:22.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.766651 26620 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
07:56:22.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.766865 26620 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/master-1/data,/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/master-1/wal
07:56:22.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "947b6ed1d6bb49e49d50a03e028cbd24"
07:56:22.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 07:56:22 on dist-test-slave-63m1"
07:56:22.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.767100 26620 fs_report.cc:389] FS layout report
07:56:22.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
07:56:22.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/master-1/wal
07:56:22.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/master-1/wal
07:56:22.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/master-1/data/data
07:56:22.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
07:56:22.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
07:56:22.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
07:56:22.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
07:56:22.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
07:56:22.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
07:56:22.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
07:56:22.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
07:56:22.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
07:56:22.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
07:56:22.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
07:56:22.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.787161 26620 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
07:56:22.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.788293 26620 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
07:56:22.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.788623 26620 kserver.cc:163] Server-wide thread pool size limit: 3276
07:56:22.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.809517 26620 rpc_server.cc:307] RPC server started. Bound to: 127.25.235.125:42159
07:56:22.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.809540 26688 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.235.125:42159 every 8 connection(s)
07:56:22.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.811340 26620 server_base.cc:1193] Dumped server information to /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/master-1/data/info.pb
07:56:22.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.814291 26541 external_mini_cluster.cc:1434] Started /tmp/dist-test-task2rU0iV/build/asan/bin/kudu as pid 26620
07:56:22.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.814462 26541 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/master-1/wal/instance
07:56:22.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.815706 26541 external_mini_cluster.cc:1372] Running /tmp/dist-test-task2rU0iV/build/asan/bin/kudu
07:56:22.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task2rU0iV/build/asan/bin/kudu
07:56:22.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/master-2/wal
07:56:22.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/master-2/data
07:56:22.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
07:56:22.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
07:56:22.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
07:56:22.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:56:22.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:56:22.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:56:22.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/master-2/logs
07:56:22.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/master-2/data/info.pb
07:56:22.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:56:22.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
07:56:22.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
07:56:22.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
07:56:22.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
07:56:22.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:56:22.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:56:22.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:56:22.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
07:56:22.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
07:56:22.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
07:56:22.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
07:56:22.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.235.124:39273
07:56:22.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.235.124
07:56:22.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
07:56:22.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.235.84:44497
07:56:22.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:56:22.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:56:22.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
07:56:22.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
07:56:22.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.25.235.126:46873,127.25.235.125:42159,127.25.235.124:39273
07:56:22.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
07:56:22.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:39705/jwks.json with env {}
07:56:22.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.816452 26689 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
07:56:22.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.826782 26689 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.235.126" port: 46873 } has no permanent_uuid. Determining permanent_uuid...
07:56:22.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.842777 26689 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.235.125" port: 42159 } has no permanent_uuid. Determining permanent_uuid...
07:56:22.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.851877 26689 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.235.124" port: 39273 } has no permanent_uuid. Determining permanent_uuid...
07:56:22.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:22.852865 26641 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.235.124:39273: connect: Connection refused (error 111)
07:56:22.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:22.854539 26689 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.235.124:39273: Network error: Client connection negotiation failed: client connection to 127.25.235.124:39273: connect: Connection refused (error 111)
07:56:22.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.903862 26689 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.235.124" port: 39273 } attempt: 1
07:56:22.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:22.906153 26689 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.235.124:39273: Network error: Client connection negotiation failed: client connection to 127.25.235.124:39273: connect: Connection refused (error 111)
07:56:22.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:22.974390 26689 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.235.124" port: 39273 } attempt: 2
07:56:22.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:22.976377 26689 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.235.124:39273: Network error: Client connection negotiation failed: client connection to 127.25.235.124:39273: connect: Connection refused (error 111)
07:56:23.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.049736 26618 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.235.125" port: 42159 } attempt: 5
07:56:23.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.056137 26618 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.235.124" port: 39273 } has no permanent_uuid. Determining permanent_uuid...
07:56:23.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:23.058007 26618 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.235.124:39273: Network error: Client connection negotiation failed: client connection to 127.25.235.124:39273: connect: Connection refused (error 111)
07:56:23.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.067631 26689 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.235.124" port: 39273 } attempt: 3
07:56:23.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:23.069876 26689 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.235.124:39273: Network error: Client connection negotiation failed: client connection to 127.25.235.124:39273: connect: Connection refused (error 111)
07:56:23.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.109270 26618 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.235.124" port: 39273 } attempt: 1
07:56:23.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:23.111644 26618 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.235.124:39273: Network error: Client connection negotiation failed: client connection to 127.25.235.124:39273: connect: Connection refused (error 111)
07:56:23.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:23.152860 26691 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
07:56:23.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:23.153203 26691 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
07:56:23.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:23.153291 26691 flags.cc:432] Enabled unsafe flag: --never_fsync=true
07:56:23.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:23.162196 26691 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
07:56:23.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:23.162299 26691 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
07:56:23.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:23.162353 26691 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
07:56:23.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:23.162393 26691 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
07:56:23.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:23.162446 26691 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
07:56:23.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:23.162479 26691 flags.cc:432] Enabled experimental flag: --jwks_url=http://localhost:39705/jwks.json
07:56:23.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.172459 26691 master_runner.cc:387] Master server non-default flags:
07:56:23.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:56:23.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.235.84:44497
07:56:23.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:56:23.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
07:56:23.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/master-2/data
07:56:23.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/master-2/wal
07:56:23.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
07:56:23.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.25.235.126:46873,127.25.235.125:42159,127.25.235.124:39273
07:56:23.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:56:23.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:56:23.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
07:56:23.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.235.124:39273
07:56:23.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
07:56:23.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
07:56:23.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
07:56:23.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:39705/jwks.json
07:56:23.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:56:23.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:56:23.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/master-2/data/info.pb
07:56:23.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.235.124
07:56:23.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
07:56:23.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
07:56:23.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:56:23.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
07:56:23.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
07:56:23.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:56:23.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/master-2/logs
07:56:23.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:56:23.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
07:56:23.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
07:56:23.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
07:56:23.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
07:56:23.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
07:56:23.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
07:56:23.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 07:43:22 UTC on bdcb31816ec0
07:56:23.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12770
07:56:23.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
07:56:23.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.174693 26691 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
07:56:23.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.177114 26691 file_cache.cc:504] Constructed file cache file cache with capacity 419430
07:56:23.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.179888 26618 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.235.124" port: 39273 } attempt: 2
07:56:23.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:23.182053 26618 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.235.124:39273: Network error: Client connection negotiation failed: client connection to 127.25.235.124:39273: connect: Connection refused (error 111)
07:56:23.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:23.186513 26700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:23.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:23.186491 26699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:23.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:23.186571 26702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:23.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.187690 26691 server_base.cc:1061] running on GCE node
07:56:23.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.188402 26691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
07:56:23.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.189828 26691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
07:56:23.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.191018 26691 hybrid_clock.cc:648] HybridClock initialized: now 1783497383190952 us; error 67 us; skew 500 ppm
07:56:23.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.191319 26691 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
07:56:23.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.193536 26691 webserver.cc:533] Webserver started at http://127.25.235.124:44697/ using document root <none> and password file <none>
07:56:23.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.194205 26691 fs_manager.cc:362] Metadata directory not provided
07:56:23.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.194300 26691 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
07:56:23.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.194581 26691 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
07:56:23.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.196805 26691 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/master-2/data/instance:
07:56:23.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "c100d5f546314da4a0df1ef31e8f4b2c"
07:56:23.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 07:56:23 on dist-test-slave-63m1"
07:56:23.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.197628 26691 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/master-2/wal/instance:
07:56:23.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "c100d5f546314da4a0df1ef31e8f4b2c"
07:56:23.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 07:56:23 on dist-test-slave-63m1"
07:56:23.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.203051 26691 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
07:56:23.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.207033 26708 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
07:56:23.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.208550 26691 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
07:56:23.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.208695 26691 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/master-2/data,/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/master-2/wal
07:56:23.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "c100d5f546314da4a0df1ef31e8f4b2c"
07:56:23.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 07:56:23 on dist-test-slave-63m1"
07:56:23.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.208858 26691 fs_report.cc:389] FS layout report
07:56:23.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
07:56:23.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/master-2/wal
07:56:23.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/master-2/wal
07:56:23.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/master-2/data/data
07:56:23.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
07:56:23.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
07:56:23.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
07:56:23.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
07:56:23.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
07:56:23.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
07:56:23.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
07:56:23.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
07:56:23.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
07:56:23.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
07:56:23.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
07:56:23.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.213084 26689 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.235.124" port: 39273 } attempt: 4
07:56:23.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:23.215240 26689 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.235.124:39273: Network error: Client connection negotiation failed: client connection to 127.25.235.124:39273: connect: Connection refused (error 111)
07:56:23.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.236052 26691 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
07:56:23.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.237210 26691 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
07:56:23.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.237563 26691 kserver.cc:163] Server-wide thread pool size limit: 3276
07:56:23.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.258610 26691 rpc_server.cc:307] RPC server started. Bound to: 127.25.235.124:39273
07:56:23.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.258627 26759 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.235.124:39273 every 8 connection(s)
07:56:23.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.260452 26691 server_base.cc:1193] Dumped server information to /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/master-2/data/info.pb
07:56:23.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.264034 26541 external_mini_cluster.cc:1434] Started /tmp/dist-test-task2rU0iV/build/asan/bin/kudu as pid 26691
07:56:23.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.264221 26541 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/master-2/wal/instance
07:56:23.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.265226 26760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
07:56:23.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.272742 26760 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.235.126" port: 46873 } has no permanent_uuid. Determining permanent_uuid...
07:56:23.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.287398 26760 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.235.125" port: 42159 } has no permanent_uuid. Determining permanent_uuid...
07:56:23.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.288277 26618 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.235.124" port: 39273 } attempt: 3
07:56:23.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.296907 26760 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.235.124" port: 39273 } has no permanent_uuid. Determining permanent_uuid...
07:56:23.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.303884 26618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b30b8ecb407419c96af767f8052e5ce: Bootstrap starting.
07:56:23.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.308144 26618 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b30b8ecb407419c96af767f8052e5ce: Neither blocks nor log segments found. Creating new log.
07:56:23.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.308365 26760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c100d5f546314da4a0df1ef31e8f4b2c: Bootstrap starting.
07:56:23.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.309525 26618 log.cc:826] T 00000000000000000000000000000000 P 8b30b8ecb407419c96af767f8052e5ce: Log is configured to *not* fsync() on all Append() calls
07:56:23.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.311414 26760 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c100d5f546314da4a0df1ef31e8f4b2c: Neither blocks nor log segments found. Creating new log.
07:56:23.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.312355 26760 log.cc:826] T 00000000000000000000000000000000 P c100d5f546314da4a0df1ef31e8f4b2c: Log is configured to *not* fsync() on all Append() calls
07:56:23.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.312397 26618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b30b8ecb407419c96af767f8052e5ce: No bootstrap required, opened a new log
07:56:23.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.315605 26760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c100d5f546314da4a0df1ef31e8f4b2c: No bootstrap required, opened a new log
07:56:23.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.320976 26618 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8b30b8ecb407419c96af767f8052e5ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b30b8ecb407419c96af767f8052e5ce" member_type: VOTER last_known_addr { host: "127.25.235.126" port: 46873 } } peers { permanent_uuid: "947b6ed1d6bb49e49d50a03e028cbd24" member_type: VOTER last_known_addr { host: "127.25.235.125" port: 42159 } } peers { permanent_uuid: "c100d5f546314da4a0df1ef31e8f4b2c" member_type: VOTER last_known_addr { host: "127.25.235.124" port: 39273 } }
07:56:23.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.321344 26618 raft_consensus.cc:385] T 00000000000000000000000000000000 P 8b30b8ecb407419c96af767f8052e5ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
07:56:23.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.321463 26618 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8b30b8ecb407419c96af767f8052e5ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b30b8ecb407419c96af767f8052e5ce, State: Initialized, Role: FOLLOWER
07:56:23.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.322234 26618 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b30b8ecb407419c96af767f8052e5ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b30b8ecb407419c96af767f8052e5ce" member_type: VOTER last_known_addr { host: "127.25.235.126" port: 46873 } } peers { permanent_uuid: "947b6ed1d6bb49e49d50a03e028cbd24" member_type: VOTER last_known_addr { host: "127.25.235.125" port: 42159 } } peers { permanent_uuid: "c100d5f546314da4a0df1ef31e8f4b2c" member_type: VOTER last_known_addr { host: "127.25.235.124" port: 39273 } }
07:56:23.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.323508 26769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b30b8ecb407419c96af767f8052e5ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b30b8ecb407419c96af767f8052e5ce" member_type: VOTER last_known_addr { host: "127.25.235.126" port: 46873 } } peers { permanent_uuid: "947b6ed1d6bb49e49d50a03e028cbd24" member_type: VOTER last_known_addr { host: "127.25.235.125" port: 42159 } } peers { permanent_uuid: "c100d5f546314da4a0df1ef31e8f4b2c" member_type: VOTER last_known_addr { host: "127.25.235.124" port: 39273 } } }
07:56:23.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.323848 26760 raft_consensus.cc:359] T 00000000000000000000000000000000 P c100d5f546314da4a0df1ef31e8f4b2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b30b8ecb407419c96af767f8052e5ce" member_type: VOTER last_known_addr { host: "127.25.235.126" port: 46873 } } peers { permanent_uuid: "947b6ed1d6bb49e49d50a03e028cbd24" member_type: VOTER last_known_addr { host: "127.25.235.125" port: 42159 } } peers { permanent_uuid: "c100d5f546314da4a0df1ef31e8f4b2c" member_type: VOTER last_known_addr { host: "127.25.235.124" port: 39273 } }
07:56:23.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.324219 26760 raft_consensus.cc:385] T 00000000000000000000000000000000 P c100d5f546314da4a0df1ef31e8f4b2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
07:56:23.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.324347 26760 raft_consensus.cc:740] T 00000000000000000000000000000000 P c100d5f546314da4a0df1ef31e8f4b2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c100d5f546314da4a0df1ef31e8f4b2c, State: Initialized, Role: FOLLOWER
07:56:23.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.324959 26618 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8b30b8ecb407419c96af767f8052e5ce [sys.catalog]: configured and running, proceeding with master startup.
07:56:23.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.325126 26760 consensus_queue.cc:260] T 00000000000000000000000000000000 P c100d5f546314da4a0df1ef31e8f4b2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b30b8ecb407419c96af767f8052e5ce" member_type: VOTER last_known_addr { host: "127.25.235.126" port: 46873 } } peers { permanent_uuid: "947b6ed1d6bb49e49d50a03e028cbd24" member_type: VOTER last_known_addr { host: "127.25.235.125" port: 42159 } } peers { permanent_uuid: "c100d5f546314da4a0df1ef31e8f4b2c" member_type: VOTER last_known_addr { host: "127.25.235.124" port: 39273 } }
07:56:23.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.325333 26769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b30b8ecb407419c96af767f8052e5ce [sys.catalog]: This master's current role is: FOLLOWER
07:56:23.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.326514 26770 sys_catalog.cc:455] T 00000000000000000000000000000000 P c100d5f546314da4a0df1ef31e8f4b2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b30b8ecb407419c96af767f8052e5ce" member_type: VOTER last_known_addr { host: "127.25.235.126" port: 46873 } } peers { permanent_uuid: "947b6ed1d6bb49e49d50a03e028cbd24" member_type: VOTER last_known_addr { host: "127.25.235.125" port: 42159 } } peers { permanent_uuid: "c100d5f546314da4a0df1ef31e8f4b2c" member_type: VOTER last_known_addr { host: "127.25.235.124" port: 39273 } } }
07:56:23.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.326824 26770 sys_catalog.cc:458] T 00000000000000000000000000000000 P c100d5f546314da4a0df1ef31e8f4b2c [sys.catalog]: This master's current role is: FOLLOWER
07:56:23.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.327518 26760 sys_catalog.cc:565] T 00000000000000000000000000000000 P c100d5f546314da4a0df1ef31e8f4b2c [sys.catalog]: configured and running, proceeding with master startup.
07:56:23.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:23.327600 26572 tablet.cc:2406] T 00000000000000000000000000000000 P 8b30b8ecb407419c96af767f8052e5ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
07:56:23.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:23.342247 26792 catalog_manager.cc:1594] T 00000000000000000000000000000000 P c100d5f546314da4a0df1ef31e8f4b2c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
07:56:23.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:23.342970 26792 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
07:56:23.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.341647 26691 master_runner.cc:187] Error getting master registration for 127.25.235.125:42159: OK, instance_id { permanent_uuid: "947b6ed1d6bb49e49d50a03e028cbd24" instance_seqno: 1783497382804010 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
07:56:23.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.342247 26552 master_runner.cc:187] Error getting master registration for 127.25.235.125:42159: OK, instance_id { permanent_uuid: "947b6ed1d6bb49e49d50a03e028cbd24" instance_seqno: 1783497382804010 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
07:56:23.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.348469 26552 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
07:56:23.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.349170 26691 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
07:56:23.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:23.356271 26790 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 8b30b8ecb407419c96af767f8052e5ce: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
07:56:23.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:23.361034 26790 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
07:56:23.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.433287 26770 raft_consensus.cc:493] T 00000000000000000000000000000000 P c100d5f546314da4a0df1ef31e8f4b2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
07:56:23.435 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.433622 26770 raft_consensus.cc:515] T 00000000000000000000000000000000 P c100d5f546314da4a0df1ef31e8f4b2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b30b8ecb407419c96af767f8052e5ce" member_type: VOTER last_known_addr { host: "127.25.235.126" port: 46873 } } peers { permanent_uuid: "947b6ed1d6bb49e49d50a03e028cbd24" member_type: VOTER last_known_addr { host: "127.25.235.125" port: 42159 } } peers { permanent_uuid: "c100d5f546314da4a0df1ef31e8f4b2c" member_type: VOTER last_known_addr { host: "127.25.235.124" port: 39273 } }
07:56:23.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.437894 26593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c100d5f546314da4a0df1ef31e8f4b2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b30b8ecb407419c96af767f8052e5ce" is_pre_election: true
07:56:23.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.436793 26770 leader_election.cc:290] T 00000000000000000000000000000000 P c100d5f546314da4a0df1ef31e8f4b2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b30b8ecb407419c96af767f8052e5ce (127.25.235.126:46873), 947b6ed1d6bb49e49d50a03e028cbd24 (127.25.235.125:42159)
07:56:23.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.438452 26593 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 8b30b8ecb407419c96af767f8052e5ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c100d5f546314da4a0df1ef31e8f4b2c in term 0.
07:56:23.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.439116 26710 leader_election.cc:304] T 00000000000000000000000000000000 P c100d5f546314da4a0df1ef31e8f4b2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b30b8ecb407419c96af767f8052e5ce, c100d5f546314da4a0df1ef31e8f4b2c; no voters: 
07:56:23.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.439600 26770 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c100d5f546314da4a0df1ef31e8f4b2c [term 0 FOLLOWER]: Leader pre-election won for term 1
07:56:23.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.439772 26770 raft_consensus.cc:493] T 00000000000000000000000000000000 P c100d5f546314da4a0df1ef31e8f4b2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
07:56:23.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.439877 26770 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c100d5f546314da4a0df1ef31e8f4b2c [term 0 FOLLOWER]: Advancing to term 1
07:56:23.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.448518 26770 raft_consensus.cc:515] T 00000000000000000000000000000000 P c100d5f546314da4a0df1ef31e8f4b2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b30b8ecb407419c96af767f8052e5ce" member_type: VOTER last_known_addr { host: "127.25.235.126" port: 46873 } } peers { permanent_uuid: "947b6ed1d6bb49e49d50a03e028cbd24" member_type: VOTER last_known_addr { host: "127.25.235.125" port: 42159 } } peers { permanent_uuid: "c100d5f546314da4a0df1ef31e8f4b2c" member_type: VOTER last_known_addr { host: "127.25.235.124" port: 39273 } }
07:56:23.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.449844 26593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c100d5f546314da4a0df1ef31e8f4b2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b30b8ecb407419c96af767f8052e5ce"
07:56:23.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.450114 26593 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8b30b8ecb407419c96af767f8052e5ce [term 0 FOLLOWER]: Advancing to term 1
07:56:23.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.454074 26593 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 8b30b8ecb407419c96af767f8052e5ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c100d5f546314da4a0df1ef31e8f4b2c in term 1.
07:56:23.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.454674 26710 leader_election.cc:304] T 00000000000000000000000000000000 P c100d5f546314da4a0df1ef31e8f4b2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b30b8ecb407419c96af767f8052e5ce, c100d5f546314da4a0df1ef31e8f4b2c; no voters: 
07:56:23.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.460760 26770 leader_election.cc:290] T 00000000000000000000000000000000 P c100d5f546314da4a0df1ef31e8f4b2c [CANDIDATE]: Term 1 election: Requested vote from peers 8b30b8ecb407419c96af767f8052e5ce (127.25.235.126:46873), 947b6ed1d6bb49e49d50a03e028cbd24 (127.25.235.125:42159)
07:56:23.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.461094 26770 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c100d5f546314da4a0df1ef31e8f4b2c [term 1 FOLLOWER]: Leader election won for term 1
07:56:23.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.461268 26770 raft_consensus.cc:697] T 00000000000000000000000000000000 P c100d5f546314da4a0df1ef31e8f4b2c [term 1 LEADER]: Becoming Leader. State: Replica: c100d5f546314da4a0df1ef31e8f4b2c, State: Running, Role: LEADER
07:56:23.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.461899 26770 consensus_queue.cc:237] T 00000000000000000000000000000000 P c100d5f546314da4a0df1ef31e8f4b2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b30b8ecb407419c96af767f8052e5ce" member_type: VOTER last_known_addr { host: "127.25.235.126" port: 46873 } } peers { permanent_uuid: "947b6ed1d6bb49e49d50a03e028cbd24" member_type: VOTER last_known_addr { host: "127.25.235.125" port: 42159 } } peers { permanent_uuid: "c100d5f546314da4a0df1ef31e8f4b2c" member_type: VOTER last_known_addr { host: "127.25.235.124" port: 39273 } }
07:56:23.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.475982 26795 sys_catalog.cc:455] T 00000000000000000000000000000000 P c100d5f546314da4a0df1ef31e8f4b2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c100d5f546314da4a0df1ef31e8f4b2c. Latest consensus state: current_term: 1 leader_uuid: "c100d5f546314da4a0df1ef31e8f4b2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b30b8ecb407419c96af767f8052e5ce" member_type: VOTER last_known_addr { host: "127.25.235.126" port: 46873 } } peers { permanent_uuid: "947b6ed1d6bb49e49d50a03e028cbd24" member_type: VOTER last_known_addr { host: "127.25.235.125" port: 42159 } } peers { permanent_uuid: "c100d5f546314da4a0df1ef31e8f4b2c" member_type: VOTER last_known_addr { host: "127.25.235.124" port: 39273 } } }
07:56:23.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.477842 26795 sys_catalog.cc:458] T 00000000000000000000000000000000 P c100d5f546314da4a0df1ef31e8f4b2c [sys.catalog]: This master's current role is: LEADER
07:56:23.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.484917 26797 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
07:56:23.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.487202 26797 catalog_manager.cc:1520] Initializing Kudu cluster ID...
07:56:23.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:23.489765 26714 tablet.cc:2406] T 00000000000000000000000000000000 P c100d5f546314da4a0df1ef31e8f4b2c: Can't schedule compaction. Clean time has not been advanced past its initial value.
07:56:23.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.514454 26689 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.235.124" port: 39273 } attempt: 5
07:56:23.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.533267 26689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 947b6ed1d6bb49e49d50a03e028cbd24: Bootstrap starting.
07:56:23.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.533411 26593 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 8b30b8ecb407419c96af767f8052e5ce [term 1 FOLLOWER]: Refusing update from remote peer c100d5f546314da4a0df1ef31e8f4b2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
07:56:23.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.534832 26770 consensus_queue.cc:1048] T 00000000000000000000000000000000 P c100d5f546314da4a0df1ef31e8f4b2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b30b8ecb407419c96af767f8052e5ce" member_type: VOTER last_known_addr { host: "127.25.235.126" port: 46873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
07:56:23.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.540148 26689 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 947b6ed1d6bb49e49d50a03e028cbd24: Neither blocks nor log segments found. Creating new log.
07:56:23.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.543359 26689 log.cc:826] T 00000000000000000000000000000000 P 947b6ed1d6bb49e49d50a03e028cbd24: Log is configured to *not* fsync() on all Append() calls
07:56:23.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.558684 26769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b30b8ecb407419c96af767f8052e5ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader c100d5f546314da4a0df1ef31e8f4b2c. Latest consensus state: current_term: 1 leader_uuid: "c100d5f546314da4a0df1ef31e8f4b2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b30b8ecb407419c96af767f8052e5ce" member_type: VOTER last_known_addr { host: "127.25.235.126" port: 46873 } } peers { permanent_uuid: "947b6ed1d6bb49e49d50a03e028cbd24" member_type: VOTER last_known_addr { host: "127.25.235.125" port: 42159 } } peers { permanent_uuid: "c100d5f546314da4a0df1ef31e8f4b2c" member_type: VOTER last_known_addr { host: "127.25.235.124" port: 39273 } } }
07:56:23.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.559028 26769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b30b8ecb407419c96af767f8052e5ce [sys.catalog]: This master's current role is: FOLLOWER
07:56:23.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.562503 26689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 947b6ed1d6bb49e49d50a03e028cbd24: No bootstrap required, opened a new log
07:56:23.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.563217 26798 mvcc.cc:204] Tried to move back new op lower bound from 7305205282881466368 to 7305205282666496000. Current Snapshot: MvccSnapshot[applied={T|T < 7305205282881466368}]
07:56:23.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.565037 26769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b30b8ecb407419c96af767f8052e5ce [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c100d5f546314da4a0df1ef31e8f4b2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b30b8ecb407419c96af767f8052e5ce" member_type: VOTER last_known_addr { host: "127.25.235.126" port: 46873 } } peers { permanent_uuid: "947b6ed1d6bb49e49d50a03e028cbd24" member_type: VOTER last_known_addr { host: "127.25.235.125" port: 42159 } } peers { permanent_uuid: "c100d5f546314da4a0df1ef31e8f4b2c" member_type: VOTER last_known_addr { host: "127.25.235.124" port: 39273 } } }
07:56:23.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.565253 26769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b30b8ecb407419c96af767f8052e5ce [sys.catalog]: This master's current role is: FOLLOWER
07:56:23.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.576669 26795 sys_catalog.cc:455] T 00000000000000000000000000000000 P c100d5f546314da4a0df1ef31e8f4b2c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c100d5f546314da4a0df1ef31e8f4b2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b30b8ecb407419c96af767f8052e5ce" member_type: VOTER last_known_addr { host: "127.25.235.126" port: 46873 } } peers { permanent_uuid: "947b6ed1d6bb49e49d50a03e028cbd24" member_type: VOTER last_known_addr { host: "127.25.235.125" port: 42159 } } peers { permanent_uuid: "c100d5f546314da4a0df1ef31e8f4b2c" member_type: VOTER last_known_addr { host: "127.25.235.124" port: 39273 } } }
07:56:23.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.576959 26795 sys_catalog.cc:458] T 00000000000000000000000000000000 P c100d5f546314da4a0df1ef31e8f4b2c [sys.catalog]: This master's current role is: LEADER
07:56:23.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.577507 26799 mvcc.cc:204] Tried to move back new op lower bound from 7305205282881466368 to 7305205282666496000. Current Snapshot: MvccSnapshot[applied={T|T < 7305205282881466368 or (T in {7305205282881466368})}]
07:56:23.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.583997 26797 catalog_manager.cc:1383] Generated new cluster ID: cab1e75c15ca49ce9fa6001dd92612d8
07:56:23.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.584142 26797 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
07:56:23.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.602936 26689 raft_consensus.cc:359] T 00000000000000000000000000000000 P 947b6ed1d6bb49e49d50a03e028cbd24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b30b8ecb407419c96af767f8052e5ce" member_type: VOTER last_known_addr { host: "127.25.235.126" port: 46873 } } peers { permanent_uuid: "947b6ed1d6bb49e49d50a03e028cbd24" member_type: VOTER last_known_addr { host: "127.25.235.125" port: 42159 } } peers { permanent_uuid: "c100d5f546314da4a0df1ef31e8f4b2c" member_type: VOTER last_known_addr { host: "127.25.235.124" port: 39273 } }
07:56:23.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.603365 26689 raft_consensus.cc:385] T 00000000000000000000000000000000 P 947b6ed1d6bb49e49d50a03e028cbd24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
07:56:23.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.603515 26689 raft_consensus.cc:740] T 00000000000000000000000000000000 P 947b6ed1d6bb49e49d50a03e028cbd24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 947b6ed1d6bb49e49d50a03e028cbd24, State: Initialized, Role: FOLLOWER
07:56:23.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.604377 26689 consensus_queue.cc:260] T 00000000000000000000000000000000 P 947b6ed1d6bb49e49d50a03e028cbd24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b30b8ecb407419c96af767f8052e5ce" member_type: VOTER last_known_addr { host: "127.25.235.126" port: 46873 } } peers { permanent_uuid: "947b6ed1d6bb49e49d50a03e028cbd24" member_type: VOTER last_known_addr { host: "127.25.235.125" port: 42159 } } peers { permanent_uuid: "c100d5f546314da4a0df1ef31e8f4b2c" member_type: VOTER last_known_addr { host: "127.25.235.124" port: 39273 } }
07:56:23.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.609966 26804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 947b6ed1d6bb49e49d50a03e028cbd24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b30b8ecb407419c96af767f8052e5ce" member_type: VOTER last_known_addr { host: "127.25.235.126" port: 46873 } } peers { permanent_uuid: "947b6ed1d6bb49e49d50a03e028cbd24" member_type: VOTER last_known_addr { host: "127.25.235.125" port: 42159 } } peers { permanent_uuid: "c100d5f546314da4a0df1ef31e8f4b2c" member_type: VOTER last_known_addr { host: "127.25.235.124" port: 39273 } } }
07:56:23.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.611366 26804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 947b6ed1d6bb49e49d50a03e028cbd24 [sys.catalog]: This master's current role is: FOLLOWER
07:56:23.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.618445 26689 sys_catalog.cc:565] T 00000000000000000000000000000000 P 947b6ed1d6bb49e49d50a03e028cbd24 [sys.catalog]: configured and running, proceeding with master startup.
07:56:23.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.619372 26662 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 947b6ed1d6bb49e49d50a03e028cbd24 [term 0 FOLLOWER]: Advancing to term 1
07:56:23.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.623356 26662 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 947b6ed1d6bb49e49d50a03e028cbd24 [term 1 FOLLOWER]: Refusing update from remote peer c100d5f546314da4a0df1ef31e8f4b2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
07:56:23.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.623814 26664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c100d5f546314da4a0df1ef31e8f4b2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "947b6ed1d6bb49e49d50a03e028cbd24" is_pre_election: true
07:56:23.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.624099 26663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c100d5f546314da4a0df1ef31e8f4b2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "947b6ed1d6bb49e49d50a03e028cbd24"
07:56:23.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.634100 26795 consensus_queue.cc:1048] T 00000000000000000000000000000000 P c100d5f546314da4a0df1ef31e8f4b2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "947b6ed1d6bb49e49d50a03e028cbd24" member_type: VOTER last_known_addr { host: "127.25.235.125" port: 42159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
07:56:23.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.641076 26797 catalog_manager.cc:1406] Generated new certificate authority record
07:56:23.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.642789 26797 catalog_manager.cc:1540] Loading token signing keys...
07:56:23.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.688664 26804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 947b6ed1d6bb49e49d50a03e028cbd24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c100d5f546314da4a0df1ef31e8f4b2c. Latest consensus state: current_term: 1 leader_uuid: "c100d5f546314da4a0df1ef31e8f4b2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b30b8ecb407419c96af767f8052e5ce" member_type: VOTER last_known_addr { host: "127.25.235.126" port: 46873 } } peers { permanent_uuid: "947b6ed1d6bb49e49d50a03e028cbd24" member_type: VOTER last_known_addr { host: "127.25.235.125" port: 42159 } } peers { permanent_uuid: "c100d5f546314da4a0df1ef31e8f4b2c" member_type: VOTER last_known_addr { host: "127.25.235.124" port: 39273 } } }
07:56:23.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.689440 26804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 947b6ed1d6bb49e49d50a03e028cbd24 [sys.catalog]: This master's current role is: FOLLOWER
07:56:23.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.690057 26809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 947b6ed1d6bb49e49d50a03e028cbd24 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c100d5f546314da4a0df1ef31e8f4b2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b30b8ecb407419c96af767f8052e5ce" member_type: VOTER last_known_addr { host: "127.25.235.126" port: 46873 } } peers { permanent_uuid: "947b6ed1d6bb49e49d50a03e028cbd24" member_type: VOTER last_known_addr { host: "127.25.235.125" port: 42159 } } peers { permanent_uuid: "c100d5f546314da4a0df1ef31e8f4b2c" member_type: VOTER last_known_addr { host: "127.25.235.124" port: 39273 } } }
07:56:23.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.690562 26809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 947b6ed1d6bb49e49d50a03e028cbd24 [sys.catalog]: This master's current role is: FOLLOWER
07:56:23.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.691923 26770 sys_catalog.cc:455] T 00000000000000000000000000000000 P c100d5f546314da4a0df1ef31e8f4b2c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c100d5f546314da4a0df1ef31e8f4b2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b30b8ecb407419c96af767f8052e5ce" member_type: VOTER last_known_addr { host: "127.25.235.126" port: 46873 } } peers { permanent_uuid: "947b6ed1d6bb49e49d50a03e028cbd24" member_type: VOTER last_known_addr { host: "127.25.235.125" port: 42159 } } peers { permanent_uuid: "c100d5f546314da4a0df1ef31e8f4b2c" member_type: VOTER last_known_addr { host: "127.25.235.124" port: 39273 } } }
07:56:23.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.692178 26770 sys_catalog.cc:458] T 00000000000000000000000000000000 P c100d5f546314da4a0df1ef31e8f4b2c [sys.catalog]: This master's current role is: LEADER
07:56:23.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.692989 26797 catalog_manager.cc:6070] T 00000000000000000000000000000000 P c100d5f546314da4a0df1ef31e8f4b2c: Generated new TSK 0
07:56:23.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.694084 26797 catalog_manager.cc:1550] Initializing in-progress tserver states...
07:56:23.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.716466 26541 external_mini_cluster.cc:1372] Running /tmp/dist-test-task2rU0iV/build/asan/bin/kudu
07:56:23.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task2rU0iV/build/asan/bin/kudu
07:56:23.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-0/wal
07:56:23.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-0/data
07:56:23.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
07:56:23.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
07:56:23.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
07:56:23.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:56:23.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:56:23.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:56:23.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-0/logs
07:56:23.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-0/data/info.pb
07:56:23.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:56:23.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
07:56:23.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
07:56:23.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
07:56:23.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
07:56:23.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:56:23.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:56:23.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:56:23.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
07:56:23.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
07:56:23.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.235.65:0
07:56:23.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.25.235.65
07:56:23.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.235.65
07:56:23.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
07:56:23.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.25.235.126:46873,127.25.235.125:42159,127.25.235.124:39273
07:56:23.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.235.84:44497
07:56:23.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:56:23.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:56:23.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
07:56:23.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.801008 26821 catalog_manager.cc:1295] Loaded cluster ID: cab1e75c15ca49ce9fa6001dd92612d8
07:56:23.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.801689 26821 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 947b6ed1d6bb49e49d50a03e028cbd24: loading cluster ID for follower catalog manager: success
07:56:23.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.805796 26821 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 947b6ed1d6bb49e49d50a03e028cbd24: acquiring CA information for follower catalog manager: success
07:56:23.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:23.808526 26821 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 947b6ed1d6bb49e49d50a03e028cbd24: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
07:56:24.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:24.177177 26817 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
07:56:24.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:24.177525 26817 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
07:56:24.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:24.177642 26817 flags.cc:432] Enabled unsafe flag: --never_fsync=true
07:56:24.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:24.186511 26817 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
07:56:24.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:24.186729 26817 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.235.65
07:56:24.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.197377 26817 tablet_server_runner.cc:78] Tablet server non-default flags:
07:56:24.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:56:24.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.235.84:44497
07:56:24.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:56:24.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
07:56:24.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-0/data
07:56:24.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-0/wal
07:56:24.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:56:24.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:56:24.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.235.65:0
07:56:24.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
07:56:24.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:56:24.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:56:24.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-0/data/info.pb
07:56:24.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.235.65
07:56:24.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
07:56:24.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.25.235.126:46873,127.25.235.125:42159,127.25.235.124:39273
07:56:24.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
07:56:24.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:56:24.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
07:56:24.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
07:56:24.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:56:24.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.25.235.65
07:56:24.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-0/logs
07:56:24.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:56:24.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
07:56:24.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
07:56:24.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
07:56:24.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
07:56:24.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
07:56:24.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
07:56:24.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 07:43:22 UTC on bdcb31816ec0
07:56:24.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12770
07:56:24.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
07:56:24.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.199622 26817 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
07:56:24.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.202224 26817 file_cache.cc:504] Constructed file cache file cache with capacity 419430
07:56:24.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:24.214238 26827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:24.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:24.214974 26828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:24.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:24.228268 26830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:24.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.228677 26817 server_base.cc:1061] running on GCE node
07:56:24.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.229696 26817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
07:56:24.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.230871 26817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
07:56:24.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.232275 26817 hybrid_clock.cc:648] HybridClock initialized: now 1783497384232170 us; error 101 us; skew 500 ppm
07:56:24.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.232681 26817 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
07:56:24.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.235193 26817 webserver.cc:533] Webserver started at http://127.25.235.65:42151/ using document root <none> and password file <none>
07:56:24.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.236104 26817 fs_manager.cc:362] Metadata directory not provided
07:56:24.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.236243 26817 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
07:56:24.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.236709 26817 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
07:56:24.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.239310 26817 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-0/data/instance:
07:56:24.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "3dd0f8e1dbda4e408e9caa288299ab5d"
07:56:24.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 07:56:24 on dist-test-slave-63m1"
07:56:24.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.240316 26817 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-0/wal/instance:
07:56:24.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "3dd0f8e1dbda4e408e9caa288299ab5d"
07:56:24.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 07:56:24 on dist-test-slave-63m1"
07:56:24.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.246189 26817 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.001s
07:56:24.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.249820 26836 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
07:56:24.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.251266 26817 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
07:56:24.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.251405 26817 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-0/data,/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-0/wal
07:56:24.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "3dd0f8e1dbda4e408e9caa288299ab5d"
07:56:24.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 07:56:24 on dist-test-slave-63m1"
07:56:24.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.251614 26817 fs_report.cc:389] FS layout report
07:56:24.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
07:56:24.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-0/wal
07:56:24.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-0/wal
07:56:24.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-0/data/data
07:56:24.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
07:56:24.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
07:56:24.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
07:56:24.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
07:56:24.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
07:56:24.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
07:56:24.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
07:56:24.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
07:56:24.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
07:56:24.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
07:56:24.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
07:56:24.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.289547 26817 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
07:56:24.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.290601 26817 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
07:56:24.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.290849 26817 kserver.cc:163] Server-wide thread pool size limit: 3276
07:56:24.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.291801 26817 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
07:56:24.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.293551 26817 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
07:56:24.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.293641 26817 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
07:56:24.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.293709 26817 ts_tablet_manager.cc:616] Registered 0 tablets
07:56:24.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.293758 26817 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
07:56:24.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.332993 26817 rpc_server.cc:307] RPC server started. Bound to: 127.25.235.65:44671
07:56:24.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.333037 26948 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.235.65:44671 every 8 connection(s)
07:56:24.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.334463 26817 server_base.cc:1193] Dumped server information to /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-0/data/info.pb
07:56:24.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.340703 26541 external_mini_cluster.cc:1434] Started /tmp/dist-test-task2rU0iV/build/asan/bin/kudu as pid 26817
07:56:24.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.340929 26541 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-0/wal/instance
07:56:24.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.354313 26541 external_mini_cluster.cc:1372] Running /tmp/dist-test-task2rU0iV/build/asan/bin/kudu
07:56:24.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task2rU0iV/build/asan/bin/kudu
07:56:24.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-1/wal
07:56:24.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-1/data
07:56:24.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
07:56:24.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
07:56:24.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
07:56:24.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:56:24.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:56:24.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:56:24.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-1/logs
07:56:24.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-1/data/info.pb
07:56:24.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:56:24.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
07:56:24.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
07:56:24.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
07:56:24.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
07:56:24.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:56:24.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:56:24.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:56:24.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
07:56:24.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
07:56:24.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.235.66:0
07:56:24.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.25.235.66
07:56:24.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.235.66
07:56:24.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
07:56:24.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.25.235.126:46873,127.25.235.125:42159,127.25.235.124:39273
07:56:24.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.235.84:44497
07:56:24.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:56:24.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:56:24.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
07:56:24.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.363888 26790 catalog_manager.cc:1295] Loaded cluster ID: cab1e75c15ca49ce9fa6001dd92612d8
07:56:24.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.364013 26790 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 8b30b8ecb407419c96af767f8052e5ce: loading cluster ID for follower catalog manager: success
07:56:24.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.367825 26790 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 8b30b8ecb407419c96af767f8052e5ce: acquiring CA information for follower catalog manager: success
07:56:24.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.370352 26790 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 8b30b8ecb407419c96af767f8052e5ce: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
07:56:24.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.378846 26950 heartbeater.cc:344] Connected to a master server at 127.25.235.126:46873
07:56:24.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.379285 26950 heartbeater.cc:461] Registering TS with master...
07:56:24.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.379864 26949 heartbeater.cc:344] Connected to a master server at 127.25.235.124:39273
07:56:24.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.379971 26949 heartbeater.cc:461] Registering TS with master...
07:56:24.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.380578 26950 heartbeater.cc:507] Master 127.25.235.126:46873 requested a full tablet report, sending...
07:56:24.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.382786 26949 heartbeater.cc:507] Master 127.25.235.124:39273 requested a full tablet report, sending...
07:56:24.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.385389 26725 ts_manager.cc:194] Registered new tserver with Master: 3dd0f8e1dbda4e408e9caa288299ab5d (127.25.235.65:44671)
07:56:24.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.386166 26582 ts_manager.cc:194] Registered new tserver with Master: 3dd0f8e1dbda4e408e9caa288299ab5d (127.25.235.65:44671)
07:56:24.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.387496 26725 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.235.65:47009
07:56:24.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.391950 26951 heartbeater.cc:344] Connected to a master server at 127.25.235.125:42159
07:56:24.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.392129 26951 heartbeater.cc:461] Registering TS with master...
07:56:24.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.392560 26951 heartbeater.cc:507] Master 127.25.235.125:42159 requested a full tablet report, sending...
07:56:24.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.394197 26653 ts_manager.cc:194] Registered new tserver with Master: 3dd0f8e1dbda4e408e9caa288299ab5d (127.25.235.65:44671)
07:56:24.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:24.725898 26957 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
07:56:24.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:24.726198 26957 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
07:56:24.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:24.726275 26957 flags.cc:432] Enabled unsafe flag: --never_fsync=true
07:56:24.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:24.734899 26957 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
07:56:24.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:24.735054 26957 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.235.66
07:56:24.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.745187 26957 tablet_server_runner.cc:78] Tablet server non-default flags:
07:56:24.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:56:24.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.235.84:44497
07:56:24.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:56:24.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
07:56:24.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-1/data
07:56:24.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-1/wal
07:56:24.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:56:24.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:56:24.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.235.66:0
07:56:24.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
07:56:24.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:56:24.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:56:24.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-1/data/info.pb
07:56:24.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.235.66
07:56:24.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
07:56:24.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.25.235.126:46873,127.25.235.125:42159,127.25.235.124:39273
07:56:24.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
07:56:24.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:56:24.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
07:56:24.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
07:56:24.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:56:24.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.25.235.66
07:56:24.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-1/logs
07:56:24.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:56:24.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
07:56:24.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
07:56:24.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
07:56:24.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
07:56:24.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
07:56:24.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
07:56:24.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 07:43:22 UTC on bdcb31816ec0
07:56:24.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12770
07:56:24.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
07:56:24.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.746824 26957 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
07:56:24.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.748502 26957 file_cache.cc:504] Constructed file cache file cache with capacity 419430
07:56:24.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:24.758324 26965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:24.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:24.758316 26964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:24.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:24.759390 26967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:24.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.759390 26957 server_base.cc:1061] running on GCE node
07:56:24.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.760036 26957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
07:56:24.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.761062 26957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
07:56:24.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.762261 26957 hybrid_clock.cc:648] HybridClock initialized: now 1783497384762207 us; error 59 us; skew 500 ppm
07:56:24.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.762615 26957 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
07:56:24.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.765076 26957 webserver.cc:533] Webserver started at http://127.25.235.66:42391/ using document root <none> and password file <none>
07:56:24.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.765841 26957 fs_manager.cc:362] Metadata directory not provided
07:56:24.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.765933 26957 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
07:56:24.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.766230 26957 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
07:56:24.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.768575 26957 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-1/data/instance:
07:56:24.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "bd0c2de15db9438f9df3c22ffa0bed6c"
07:56:24.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 07:56:24 on dist-test-slave-63m1"
07:56:24.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.769369 26957 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-1/wal/instance:
07:56:24.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "bd0c2de15db9438f9df3c22ffa0bed6c"
07:56:24.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 07:56:24 on dist-test-slave-63m1"
07:56:24.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.774833 26957 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.000s
07:56:24.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.778136 26973 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
07:56:24.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.779520 26957 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
07:56:24.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.779652 26957 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-1/data,/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-1/wal
07:56:24.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "bd0c2de15db9438f9df3c22ffa0bed6c"
07:56:24.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 07:56:24 on dist-test-slave-63m1"
07:56:24.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.779870 26957 fs_report.cc:389] FS layout report
07:56:24.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
07:56:24.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-1/wal
07:56:24.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-1/wal
07:56:24.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-1/data/data
07:56:24.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
07:56:24.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
07:56:24.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
07:56:24.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
07:56:24.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
07:56:24.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
07:56:24.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
07:56:24.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
07:56:24.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
07:56:24.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
07:56:24.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
07:56:24.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.796674 26957 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
07:56:24.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.797575 26957 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
07:56:24.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.797891 26957 kserver.cc:163] Server-wide thread pool size limit: 3276
07:56:24.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.798842 26957 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
07:56:24.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.800519 26957 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
07:56:24.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.800606 26957 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
07:56:24.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.800676 26957 ts_tablet_manager.cc:616] Registered 0 tablets
07:56:24.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.800734 26957 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
07:56:24.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.841427 26957 rpc_server.cc:307] RPC server started. Bound to: 127.25.235.66:42989
07:56:24.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.841480 27085 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.235.66:42989 every 8 connection(s)
07:56:24.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.842797 26957 server_base.cc:1193] Dumped server information to /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-1/data/info.pb
07:56:24.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.854140 26541 external_mini_cluster.cc:1434] Started /tmp/dist-test-task2rU0iV/build/asan/bin/kudu as pid 26957
07:56:24.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.854349 26541 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-1/wal/instance
07:56:24.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.859083 27087 heartbeater.cc:344] Connected to a master server at 127.25.235.126:46873
07:56:24.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.859546 27087 heartbeater.cc:461] Registering TS with master...
07:56:24.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.860401 26541 external_mini_cluster.cc:1372] Running /tmp/dist-test-task2rU0iV/build/asan/bin/kudu
07:56:24.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task2rU0iV/build/asan/bin/kudu
07:56:24.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/wal
07:56:24.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/data
07:56:24.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
07:56:24.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
07:56:24.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
07:56:24.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:56:24.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:56:24.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:56:24.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/logs
07:56:24.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/data/info.pb
07:56:24.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:56:24.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
07:56:24.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
07:56:24.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
07:56:24.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
07:56:24.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:56:24.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:56:24.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:56:24.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
07:56:24.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
07:56:24.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.235.67:0
07:56:24.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.25.235.67
07:56:24.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.235.67
07:56:24.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
07:56:24.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.25.235.126:46873,127.25.235.125:42159,127.25.235.124:39273
07:56:24.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.235.84:44497
07:56:24.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:56:24.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:56:24.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
07:56:24.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.860443 27087 heartbeater.cc:507] Master 127.25.235.126:46873 requested a full tablet report, sending...
07:56:24.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.862401 26582 ts_manager.cc:194] Registered new tserver with Master: bd0c2de15db9438f9df3c22ffa0bed6c (127.25.235.66:42989)
07:56:24.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.862756 27088 heartbeater.cc:344] Connected to a master server at 127.25.235.125:42159
07:56:24.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.862903 27088 heartbeater.cc:461] Registering TS with master...
07:56:24.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.863397 27088 heartbeater.cc:507] Master 127.25.235.125:42159 requested a full tablet report, sending...
07:56:24.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.864230 27086 heartbeater.cc:344] Connected to a master server at 127.25.235.124:39273
07:56:24.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.864355 27086 heartbeater.cc:461] Registering TS with master...
07:56:24.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.864406 26653 ts_manager.cc:194] Registered new tserver with Master: bd0c2de15db9438f9df3c22ffa0bed6c (127.25.235.66:42989)
07:56:24.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.864845 27086 heartbeater.cc:507] Master 127.25.235.124:39273 requested a full tablet report, sending...
07:56:24.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.865864 26725 ts_manager.cc:194] Registered new tserver with Master: bd0c2de15db9438f9df3c22ffa0bed6c (127.25.235.66:42989)
07:56:24.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:24.866698 26725 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.235.66:42901
07:56:25.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:25.193456 27093 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
07:56:25.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:25.193763 27093 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
07:56:25.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:25.193841 27093 flags.cc:432] Enabled unsafe flag: --never_fsync=true
07:56:25.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:25.202425 27093 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
07:56:25.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:25.202586 27093 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.235.67
07:56:25.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:25.212443 27093 tablet_server_runner.cc:78] Tablet server non-default flags:
07:56:25.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:56:25.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.235.84:44497
07:56:25.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:56:25.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
07:56:25.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/data
07:56:25.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/wal
07:56:25.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:56:25.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:56:25.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.235.67:0
07:56:25.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
07:56:25.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:56:25.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:56:25.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/data/info.pb
07:56:25.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.235.67
07:56:25.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
07:56:25.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.25.235.126:46873,127.25.235.125:42159,127.25.235.124:39273
07:56:25.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
07:56:25.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:56:25.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
07:56:25.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
07:56:25.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:56:25.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.25.235.67
07:56:25.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/logs
07:56:25.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:56:25.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
07:56:25.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
07:56:25.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
07:56:25.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
07:56:25.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
07:56:25.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
07:56:25.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 07:43:22 UTC on bdcb31816ec0
07:56:25.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12770
07:56:25.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
07:56:25.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:25.214247 27093 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
07:56:25.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:25.216717 27093 file_cache.cc:504] Constructed file cache file cache with capacity 419430
07:56:25.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:25.228443 27098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:25.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:25.228399 27099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:25.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:25.229250 27101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:25.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:25.229898 27093 server_base.cc:1061] running on GCE node
07:56:25.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:25.230546 27093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
07:56:25.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:25.231649 27093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
07:56:25.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:25.232861 27093 hybrid_clock.cc:648] HybridClock initialized: now 1783497385232817 us; error 39 us; skew 500 ppm
07:56:25.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:25.233296 27093 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
07:56:25.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:25.235522 27093 webserver.cc:533] Webserver started at http://127.25.235.67:43721/ using document root <none> and password file <none>
07:56:25.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:25.236297 27093 fs_manager.cc:362] Metadata directory not provided
07:56:25.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:25.236433 27093 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
07:56:25.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:25.236819 27093 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
07:56:25.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:25.239351 27093 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/data/instance:
07:56:25.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "953ec1e7cb1345bd986b2b0171b6bcc2"
07:56:25.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 07:56:25 on dist-test-slave-63m1"
07:56:25.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:25.240177 27093 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/wal/instance:
07:56:25.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "953ec1e7cb1345bd986b2b0171b6bcc2"
07:56:25.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 07:56:25 on dist-test-slave-63m1"
07:56:25.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:25.245885 27093 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.001s
07:56:25.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:25.249369 27107 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
07:56:25.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:25.250895 27093 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
07:56:25.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:25.251085 27093 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/data,/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/wal
07:56:25.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "953ec1e7cb1345bd986b2b0171b6bcc2"
07:56:25.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 07:56:25 on dist-test-slave-63m1"
07:56:25.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:25.251361 27093 fs_report.cc:389] FS layout report
07:56:25.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
07:56:25.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/wal
07:56:25.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/wal
07:56:25.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/data/data
07:56:25.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
07:56:25.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
07:56:25.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
07:56:25.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
07:56:25.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
07:56:25.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
07:56:25.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
07:56:25.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
07:56:25.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
07:56:25.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
07:56:25.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
07:56:25.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:25.271700 27093 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
07:56:25.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:25.272735 27093 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
07:56:25.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:25.273105 27093 kserver.cc:163] Server-wide thread pool size limit: 3276
07:56:25.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:25.274174 27093 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
07:56:25.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:25.275878 27093 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
07:56:25.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:25.276005 27093 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
07:56:25.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:25.276108 27093 ts_tablet_manager.cc:616] Registered 0 tablets
07:56:25.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:25.276182 27093 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
07:56:25.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:25.309912 27093 rpc_server.cc:307] RPC server started. Bound to: 127.25.235.67:36093
07:56:25.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:25.309969 27219 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.235.67:36093 every 8 connection(s)
07:56:25.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:25.311460 27093 server_base.cc:1193] Dumped server information to /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/data/info.pb
07:56:25.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:25.319968 26541 external_mini_cluster.cc:1434] Started /tmp/dist-test-task2rU0iV/build/asan/bin/kudu as pid 27093
07:56:25.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:25.320163 26541 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/wal/instance
07:56:25.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:25.328512 27221 heartbeater.cc:344] Connected to a master server at 127.25.235.126:46873
07:56:25.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:25.328915 27222 heartbeater.cc:344] Connected to a master server at 127.25.235.125:42159
07:56:25.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:25.329066 27222 heartbeater.cc:461] Registering TS with master...
07:56:25.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:25.329414 27221 heartbeater.cc:461] Registering TS with master...
07:56:25.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:25.330212 27220 heartbeater.cc:344] Connected to a master server at 127.25.235.124:39273
07:56:25.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:25.330353 27222 heartbeater.cc:507] Master 127.25.235.125:42159 requested a full tablet report, sending...
07:56:25.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:25.330361 27220 heartbeater.cc:461] Registering TS with master...
07:56:25.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:25.330371 27221 heartbeater.cc:507] Master 127.25.235.126:46873 requested a full tablet report, sending...
07:56:25.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:25.330992 27220 heartbeater.cc:507] Master 127.25.235.124:39273 requested a full tablet report, sending...
07:56:25.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:25.332181 26582 ts_manager.cc:194] Registered new tserver with Master: 953ec1e7cb1345bd986b2b0171b6bcc2 (127.25.235.67:36093)
07:56:25.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:25.332226 26653 ts_manager.cc:194] Registered new tserver with Master: 953ec1e7cb1345bd986b2b0171b6bcc2 (127.25.235.67:36093)
07:56:25.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:25.332223 26725 ts_manager.cc:194] Registered new tserver with Master: 953ec1e7cb1345bd986b2b0171b6bcc2 (127.25.235.67:36093)
07:56:25.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:25.333323 26725 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.235.67:48819
07:56:25.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:25.346935 26541 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
07:56:25.350 [DEBUG - main] (MiniKuduCluster.java:195) Response: 
07:56:25.351 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_masters {
}

07:56:25.361 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_masters {
  masters {
    id {
      type: MASTER
      index: 0
    }
    bound_rpc_address {
      host: "127.25.235.126"
      port: 46873
    }
    bound_http_address {
      host: "127.25.235.126"
      port: 37817
    }
  }
  masters {
    id {
      type: MASTER
      index: 1
    }
    bound_rpc_address {
      host: "127.25.235.125"
      port: 42159
    }
    bound_http_address {
      host: "127.25.235.125"
      port: 36231
    }
  }
  masters {
    id {
      type: MASTER
      index: 2
    }
    bound_rpc_address {
      host: "127.25.235.124"
      port: 39273
    }
    bound_http_address {
      host: "127.25.235.124"
      port: 44697
    }
  }
}

07:56:25.364 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_tservers {
}

07:56:25.367 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_tservers {
  tservers {
    id {
      type: TSERVER
      index: 0
    }
    bound_rpc_address {
      host: "127.25.235.65"
      port: 44671
    }
    bound_http_address {
      host: "127.25.235.65"
      port: 42151
    }
  }
  tservers {
    id {
      type: TSERVER
      index: 1
    }
    bound_rpc_address {
      host: "127.25.235.66"
      port: 42989
    }
    bound_http_address {
      host: "127.25.235.66"
      port: 42391
    }
  }
  tservers {
    id {
      type: TSERVER
      index: 2
    }
    bound_rpc_address {
      host: "127.25.235.67"
      port: 36093
    }
    bound_http_address {
      host: "127.25.235.67"
      port: 43721
    }
  }
}

07:56:25.367 [INFO - main] (KuduTestHarness.java:157) Creating a new Kudu client...
07:56:25.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:25.390782 26949 heartbeater.cc:499] Master 127.25.235.124:39273 was elected leader, sending a full tablet report...
07:56:25.532 [DEBUG - main] (CompositeStrategy.java:59) Unable to load modern Subject methods, relying only on legacy methods
java.lang.NoSuchMethodException: javax.security.auth.Subject.current()
	at java.base/java.lang.Class.getDeclaredMethod(Class.java:2677)
	at org.apache.kudu.client.internals.ModernStrategy.<init>(ModernStrategy.java:51)
	at org.apache.kudu.client.internals.CompositeStrategy.<init>(CompositeStrategy.java:52)
	at org.apache.kudu.client.internals.CompositeStrategy.<clinit>(CompositeStrategy.java:41)
	at org.apache.kudu.client.internals.SecurityManagerCompatibility.get(SecurityManagerCompatibility.java:43)
	at org.apache.kudu.client.SecurityContext.setupSubject(SecurityContext.java:155)
	at org.apache.kudu.client.SecurityContext.<init>(SecurityContext.java:138)
	at org.apache.kudu.client.AsyncKuduClient.<init>(AsyncKuduClient.java:431)
	at org.apache.kudu.client.AsyncKuduClient$AsyncKuduClientBuilder.build(AsyncKuduClient.java:3109)
	at org.apache.kudu.test.KuduTestHarness.before(KuduTestHarness.java:161)
	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:50)
	at org.apache.kudu.test.junit.RetryRule$RetryStatement.doOneAttemptAndReport(RetryRule.java:184)
	at org.apache.kudu.test.junit.RetryRule$RetryStatement.evaluate(RetryRule.java:244)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.junit.runners.Suite.runChild(Suite.java:128)
	at org.junit.runners.Suite.runChild(Suite.java:27)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
	at org.junit.runner.JUnitCore.runMain(JUnitCore.java:77)
	at org.junit.runner.JUnitCore.main(JUnitCore.java:36)
07:56:25.587 [DEBUG - main] (SecurityUtil.java:122) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 
07:56:25.784 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.25.235.125:42159(127.25.235.125:42159)] connecting to peer
07:56:25.852 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.25.235.126:46873(127.25.235.126:46873)] connecting to peer
07:56:25.857 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.25.235.124:39273(127.25.235.124:39273)] connecting to peer
07:56:25.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:25.869356 27086 heartbeater.cc:499] Master 127.25.235.124:39273 was elected leader, sending a full tablet report...
07:56:25.871 [DEBUG - kudu-nio-2] (Connection.java:757) [peer master-127.25.235.124:39273(127.25.235.124:39273)] Successfully connected to peer
07:56:25.872 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.25.235.126:46873(127.25.235.126:46873)] Successfully connected to peer
07:56:25.895 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.25.235.125:42159(127.25.235.125:42159)] Successfully connected to peer
07:56:25.974 [DEBUG - kudu-nio-2] (Negotiator.java:528) SASL mechanism PLAIN chosen for peer 127.25.235.124
07:56:25.974 [DEBUG - kudu-nio-1] (Negotiator.java:528) SASL mechanism PLAIN chosen for peer 127.25.235.126
07:56:25.975 [DEBUG - kudu-nio-0] (Negotiator.java:528) SASL mechanism PLAIN chosen for peer 127.25.235.125
07:56:26.221 [DEBUG - kudu-nio-2] (Negotiator.java:918) Authenticated connection [id: 0xb8480209, L:/127.0.0.1:58058 - R:/127.25.235.124:39273] using SASL/PLAIN
07:56:26.221 [DEBUG - kudu-nio-0] (Negotiator.java:918) Authenticated connection [id: 0x10fc143f, L:/127.0.0.1:54824 - R:/127.25.235.125:42159] using SASL/PLAIN
07:56:26.221 [DEBUG - kudu-nio-1] (Negotiator.java:918) Authenticated connection [id: 0xc8c3e6d4, L:/127.0.0.1:43834 - R:/127.25.235.126:46873] using SASL/PLAIN
07:56:26.269 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet Kudu Master for table 'Kudu Master' with partition [<start>, <end>)
07:56:26.270 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=CreateTable, tablet=null, attempt=1, TimeoutTracker(timeout=50000, elapsed=532), Traces: [0ms] refreshing cache from master, [29ms] Sub RPC ConnectToMaster: sending RPC to server master-127.25.235.125:42159, [109ms] Sub RPC ConnectToMaster: sending RPC to server master-127.25.235.126:46873, [115ms] Sub RPC ConnectToMaster: sending RPC to server master-127.25.235.124:39273, [495ms] Sub RPC ConnectToMaster: received response from server master-127.25.235.126:46873: OK, [495ms] Sub RPC ConnectToMaster: received response from server master-127.25.235.124:39273: OK, [495ms] Sub RPC ConnectToMaster: received response from server master-127.25.235.125:42159: OK, deferred=null) after lookup
07:56:26.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.318643 26725 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58058:
07:56:26.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) name: "org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721"
07:56:26.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) schema {
07:56:26.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
07:56:26.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "key"
07:56:26.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
07:56:26.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: true
07:56:26.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
07:56:26.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
07:56:26.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
07:56:26.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
07:56:26.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
07:56:26.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
07:56:26.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column1_i"
07:56:26.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
07:56:26.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
07:56:26.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
07:56:26.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
07:56:26.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
07:56:26.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
07:56:26.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
07:56:26.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
07:56:26.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column2_i"
07:56:26.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
07:56:26.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
07:56:26.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
07:56:26.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
07:56:26.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
07:56:26.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
07:56:26.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
07:56:26.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
07:56:26.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column3_s"
07:56:26.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: STRING
07:56:26.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
07:56:26.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: true
07:56:26.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     encoding: DICT_ENCODING
07:56:26.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     compression: LZ4
07:56:26.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 4096
07:56:26.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
07:56:26.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
07:56:26.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
07:56:26.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
07:56:26.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column4_b"
07:56:26.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: BOOL
07:56:26.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
07:56:26.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
07:56:26.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
07:56:26.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
07:56:26.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
07:56:26.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
07:56:26.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) }
07:56:26.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) partition_schema {
07:56:26.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   range_schema {
07:56:26.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     columns {
07:56:26.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)       name: "key"
07:56:26.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     }
07:56:26.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
07:56:26.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) }
07:56:26.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:26.320451 26725 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
07:56:26.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.335724 27220 heartbeater.cc:499] Master 127.25.235.124:39273 was elected leader, sending a full tablet report...
07:56:26.393 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:894) Opened table 69bb0e6bdd5e45c8a6a76f2d0c604e7a
07:56:26.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.415113 26884 tablet_service.cc:1511] Processing CreateTablet for tablet 9bc0a52dc27145398e9a7b4bdd4280b5 (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721 [id=69bb0e6bdd5e45c8a6a76f2d0c604e7a]), partition=RANGE (key) PARTITION UNBOUNDED
07:56:26.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.415817 27155 tablet_service.cc:1511] Processing CreateTablet for tablet 9bc0a52dc27145398e9a7b4bdd4280b5 (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721 [id=69bb0e6bdd5e45c8a6a76f2d0c604e7a]), partition=RANGE (key) PARTITION UNBOUNDED
07:56:26.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.417184 27155 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bc0a52dc27145398e9a7b4bdd4280b5. 1 dirs total, 0 dirs full, 0 dirs failed
07:56:26.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.420684 27021 tablet_service.cc:1511] Processing CreateTablet for tablet 9bc0a52dc27145398e9a7b4bdd4280b5 (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721 [id=69bb0e6bdd5e45c8a6a76f2d0c604e7a]), partition=RANGE (key) PARTITION UNBOUNDED
07:56:26.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.422796 27021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bc0a52dc27145398e9a7b4bdd4280b5. 1 dirs total, 0 dirs full, 0 dirs failed
07:56:26.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.425995 26884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bc0a52dc27145398e9a7b4bdd4280b5. 1 dirs total, 0 dirs full, 0 dirs failed
07:56:26.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.438817 27252 tablet_bootstrap.cc:492] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2: Bootstrap starting.
07:56:26.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.443480 27252 tablet_bootstrap.cc:654] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2: Neither blocks nor log segments found. Creating new log.
07:56:26.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.454178 27252 log.cc:826] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2: Log is configured to *not* fsync() on all Append() calls
07:56:26.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.458349 27253 tablet_bootstrap.cc:492] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c: Bootstrap starting.
07:56:26.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.460908 27254 tablet_bootstrap.cc:492] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d: Bootstrap starting.
07:56:26.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.464092 27253 tablet_bootstrap.cc:654] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c: Neither blocks nor log segments found. Creating new log.
07:56:26.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.464982 27254 tablet_bootstrap.cc:654] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d: Neither blocks nor log segments found. Creating new log.
07:56:26.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.465754 27253 log.cc:826] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c: Log is configured to *not* fsync() on all Append() calls
07:56:26.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.466033 27252 tablet_bootstrap.cc:492] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2: No bootstrap required, opened a new log
07:56:26.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.466572 27252 ts_tablet_manager.cc:1403] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2: Time spent bootstrapping tablet: real 0.028s	user 0.006s	sys 0.004s
07:56:26.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.467008 27254 log.cc:826] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d: Log is configured to *not* fsync() on all Append() calls
07:56:26.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.475723 27253 tablet_bootstrap.cc:492] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c: No bootstrap required, opened a new log
07:56:26.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.475484 27252 raft_consensus.cc:359] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:26.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.475940 27252 raft_consensus.cc:385] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
07:56:26.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.476060 27252 raft_consensus.cc:740] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 953ec1e7cb1345bd986b2b0171b6bcc2, State: Initialized, Role: FOLLOWER
07:56:26.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.476222 27254 tablet_bootstrap.cc:492] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d: No bootstrap required, opened a new log
07:56:26.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.476603 27254 ts_tablet_manager.cc:1403] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d: Time spent bootstrapping tablet: real 0.016s	user 0.009s	sys 0.003s
07:56:26.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.476641 27253 ts_tablet_manager.cc:1403] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c: Time spent bootstrapping tablet: real 0.019s	user 0.007s	sys 0.008s
07:56:26.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.477051 27252 consensus_queue.cc:260] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:26.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.484754 27254 raft_consensus.cc:359] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:26.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.485586 27254 raft_consensus.cc:385] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
07:56:26.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.485853 27254 raft_consensus.cc:740] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dd0f8e1dbda4e408e9caa288299ab5d, State: Initialized, Role: FOLLOWER
07:56:26.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.486812 27254 consensus_queue.cc:260] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:26.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.491642 27253 raft_consensus.cc:359] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:26.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.492498 27253 raft_consensus.cc:385] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
07:56:26.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.492817 27253 raft_consensus.cc:740] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd0c2de15db9438f9df3c22ffa0bed6c, State: Initialized, Role: FOLLOWER
07:56:26.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.495308 27254 ts_tablet_manager.cc:1434] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d: Time spent starting tablet: real 0.018s	user 0.009s	sys 0.006s
07:56:26.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.505752 27253 consensus_queue.cc:260] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:26.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.507017 27258 raft_consensus.cc:493] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
07:56:26.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.507815 27258 raft_consensus.cc:515] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:26.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.514153 27252 ts_tablet_manager.cc:1434] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2: Time spent starting tablet: real 0.047s	user 0.016s	sys 0.011s
07:56:26.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.514494 27258 leader_election.cc:290] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3dd0f8e1dbda4e408e9caa288299ab5d (127.25.235.65:44671), bd0c2de15db9438f9df3c22ffa0bed6c (127.25.235.66:42989)
07:56:26.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.534212 27253 ts_tablet_manager.cc:1434] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c: Time spent starting tablet: real 0.057s	user 0.022s	sys 0.005s
07:56:26.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.544042 26904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bc0a52dc27145398e9a7b4bdd4280b5" candidate_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" is_pre_election: true
07:56:26.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.544116 27041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bc0a52dc27145398e9a7b4bdd4280b5" candidate_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" is_pre_election: true
07:56:26.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.544627 26904 raft_consensus.cc:2468] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 953ec1e7cb1345bd986b2b0171b6bcc2 in term 0.
07:56:26.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.544627 27041 raft_consensus.cc:2468] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 953ec1e7cb1345bd986b2b0171b6bcc2 in term 0.
07:56:26.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.545315 27109 leader_election.cc:304] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3dd0f8e1dbda4e408e9caa288299ab5d, 953ec1e7cb1345bd986b2b0171b6bcc2; no voters: 
07:56:26.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.545781 27260 raft_consensus.cc:2804] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
07:56:26.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.545938 27260 raft_consensus.cc:493] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
07:56:26.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.546073 27260 raft_consensus.cc:3060] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [term 0 FOLLOWER]: Advancing to term 1
07:56:26.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.549301 27260 raft_consensus.cc:515] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:26.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.550170 27260 leader_election.cc:290] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [CANDIDATE]: Term 1 election: Requested vote from peers 3dd0f8e1dbda4e408e9caa288299ab5d (127.25.235.65:44671), bd0c2de15db9438f9df3c22ffa0bed6c (127.25.235.66:42989)
07:56:26.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.550699 26904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bc0a52dc27145398e9a7b4bdd4280b5" candidate_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d"
07:56:26.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.550843 27041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bc0a52dc27145398e9a7b4bdd4280b5" candidate_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c"
07:56:26.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.550969 26904 raft_consensus.cc:3060] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 0 FOLLOWER]: Advancing to term 1
07:56:26.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.551029 27041 raft_consensus.cc:3060] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [term 0 FOLLOWER]: Advancing to term 1
07:56:26.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.553582 26904 raft_consensus.cc:2468] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 953ec1e7cb1345bd986b2b0171b6bcc2 in term 1.
07:56:26.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.554124 27041 raft_consensus.cc:2468] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 953ec1e7cb1345bd986b2b0171b6bcc2 in term 1.
07:56:26.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.554147 27109 leader_election.cc:304] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3dd0f8e1dbda4e408e9caa288299ab5d, 953ec1e7cb1345bd986b2b0171b6bcc2; no voters: 
07:56:26.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.554608 27260 raft_consensus.cc:2804] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [term 1 FOLLOWER]: Leader election won for term 1
07:56:26.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.554833 27260 raft_consensus.cc:697] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [term 1 LEADER]: Becoming Leader. State: Replica: 953ec1e7cb1345bd986b2b0171b6bcc2, State: Running, Role: LEADER
07:56:26.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.555418 27260 consensus_queue.cc:237] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:26.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.562450 26724 catalog_manager.cc:5697] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 953ec1e7cb1345bd986b2b0171b6bcc2 (127.25.235.67). New cstate: current_term: 1 leader_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } health_report { overall_health: UNKNOWN } } }
07:56:26.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:26.565263 27223 tablet.cc:2406] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2: Can't schedule compaction. Clean time has not been advanced past its initial value.
07:56:26.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:26.603470 27089 tablet.cc:2406] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c: Can't schedule compaction. Clean time has not been advanced past its initial value.
07:56:26.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:26.612852 26952 tablet.cc:2406] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d: Can't schedule compaction. Clean time has not been advanced past its initial value.
07:56:26.667 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:894) Opened table 69bb0e6bdd5e45c8a6a76f2d0c604e7a
07:56:26.698 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:26.703 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299995, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671),953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)[L],bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)]}]
07:56:26.704 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=null, attempt=1, sequence_id=-1, TimeoutTracker(timeout=30000, elapsed=28), Traces: [0ms] refreshing cache from master, [0ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [19ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=null) after lookup
07:56:26.713 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)] connecting to peer
07:56:26.715 [DEBUG - kudu-nio-3] (Connection.java:757) [peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)] Successfully connected to peer
07:56:26.789 [DEBUG - kudu-nio-3] (Negotiator.java:918) Authenticated connection [id: 0x0ab71f4e, L:/127.0.0.1:46598 - R:/127.25.235.67:36093] using TOKEN/null
07:56:26.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.800628 26904 raft_consensus.cc:1275] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 1 FOLLOWER]: Refusing update from remote peer 953ec1e7cb1345bd986b2b0171b6bcc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
07:56:26.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.800671 27041 raft_consensus.cc:1275] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [term 1 FOLLOWER]: Refusing update from remote peer 953ec1e7cb1345bd986b2b0171b6bcc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
07:56:26.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.801852 27258 consensus_queue.cc:1048] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
07:56:26.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.802210 27260 consensus_queue.cc:1048] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
07:56:26.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.833196 27269 mvcc.cc:204] Tried to move back new op lower bound from 7305205296323620864 to 7305205295335931904. Current Snapshot: MvccSnapshot[applied={T|T < 7305205296323620864}]
07:56:26.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.837668 27271 mvcc.cc:204] Tried to move back new op lower bound from 7305205296323620864 to 7305205295335931904. Current Snapshot: MvccSnapshot[applied={T|T < 7305205296323620864}]
07:56:26.955 [DEBUG - kudu-nio-3] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 3 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
07:56:26.957 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.25.235.67:36093
07:56:26.959 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

07:56:26.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.959990 26541 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-task2rU0iV/build/asan/bin/kudu pid 27093
07:56:26.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:26.972823 27215 generic_service.cc:196] Checking for leaks (request via RPC)
07:56:27.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:27.075706 26541 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task2rU0iV/build/asan/bin/kudu with pid 27093
07:56:27.094 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
07:56:27.096 [DEBUG - kudu-nio-3] (Connection.java:239) [peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)] handling channelInactive
07:56:27.097 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)] cleaning up while in state READY due to: connection closed
07:56:27.097 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093) for tablet 9bc0a52dc27145398e9a7b4bdd4280b5: connection closed
07:56:27.102 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:27.102 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671),953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)[L],bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)]}]
07:56:27.103 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, attempt=2, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=7), Traces: [0ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [2ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: connection closed, [2ms] delaying RPC due to: Network error: connection closed, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [7ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=Deferred@1059985019(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f19483391e8@3089969 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3a1fe13b -> wakeup thread Time-limited test)) after lookup
07:56:27.103 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)] connecting to peer
07:56:27.106 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:27.106 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)] cleaning up while in state CONNECTING due to: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:27.106 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093) for tablet 9bc0a52dc27145398e9a7b4bdd4280b5: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:27.122 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:27.122 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671),953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)[L],bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)]}]
07:56:27.123 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, attempt=4, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=27), Traces: [0ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [2ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: connection closed, [2ms] delaying RPC due to: Network error: connection closed, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [7ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [8ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [11ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [11ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [27ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=Deferred@1059985019(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f19483391e8@3089969 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@556426527 after retry RPC@895622009), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3a1fe13b -> wakeup thread Time-limited test -> (continuation of Deferred@556426527 after retry RPC@895622009))) after lookup
07:56:27.123 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)] connecting to peer
07:56:27.125 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:27.125 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)] cleaning up while in state CONNECTING due to: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:27.125 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093) for tablet 9bc0a52dc27145398e9a7b4bdd4280b5: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:27.163 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:27.163 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671),953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)[L],bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)]}]
07:56:27.163 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, attempt=6, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=68), Traces: [0ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [2ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: connection closed, [2ms] delaying RPC due to: Network error: connection closed, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [7ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [8ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [11ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [11ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [27ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [28ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [30ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [30ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [67ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=Deferred@1059985019(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f19483391e8@3089969 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@556426527 after retry RPC@895622009) -> (continuation of Deferred@89908544 after retry RPC@1501575241), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3a1fe13b -> wakeup thread Time-limited test -> (continuation of Deferred@556426527 after retry RPC@895622009) -> (continuation of Deferred@89908544 after retry RPC@1501575241))) after lookup
07:56:27.164 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)] connecting to peer
07:56:27.166 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:27.166 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)] cleaning up while in state CONNECTING due to: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:27.166 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093) for tablet 9bc0a52dc27145398e9a7b4bdd4280b5: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:27.282 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:27.283 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671),953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)[L],bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)]}]
07:56:27.283 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, attempt=8, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=188), Traces: [0ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [2ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: connection closed, [2ms] delaying RPC due to: Network error: connection closed, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [7ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [8ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [11ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [11ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [27ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [28ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [30ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [30ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [67ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [69ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [71ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [71ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [183ms] refreshing cache from master, [184ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [187ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=Deferred@1059985019(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f19483391e8@3089969 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@556426527 after retry RPC@895622009) -> (continuation of Deferred@89908544 after retry RPC@1501575241) -> (continuation of Deferred@765031220 after retry RPC@2092941877), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3a1fe13b -> wakeup thread Time-limited test -> (continuation of Deferred@556426527 after retry RPC@895622009) -> (continuation of Deferred@89908544 after retry RPC@1501575241) -> (continuation of Deferred@765031220 after retry RPC@2092941877))) after lookup
07:56:27.284 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)] connecting to peer
07:56:27.286 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:27.286 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)] cleaning up while in state CONNECTING due to: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:27.286 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093) for tablet 9bc0a52dc27145398e9a7b4bdd4280b5: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:27.663 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:27.663 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671),953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)[L],bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)]}]
07:56:27.664 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, attempt=10, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=568), Traces: [0ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [2ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: connection closed, [2ms] delaying RPC due to: Network error: connection closed, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [7ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [8ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [11ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [11ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [27ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [28ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [30ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [30ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [67ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [69ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [71ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [71ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [183ms] refreshing cache from master, [184ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [187ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [188ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [191ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [191ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [563ms] refreshing cache from master, [564ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [567ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=Deferred@1059985019(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f19483391e8@3089969 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@556426527 after retry RPC@895622009) -> (continuation of Deferred@89908544 after retry RPC@1501575241) -> (continuation of Deferred@765031220 after retry RPC@2092941877) -> (continuation of Deferred@517141542 after retry RPC@1109700985), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3a1fe13b -> wakeup thread Time-limited test -> (continuation of Deferred@556426527 after retry RPC@895622009) -> (continuation of Deferred@89908544 after retry RPC@1501575241) -> (continuation of Deferred@765031220 after retry RPC@2092941877) -> (continuation of Deferred@517141542 after retry RPC@1109700985))) after lookup
07:56:27.665 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)] connecting to peer
07:56:27.666 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:27.666 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)] cleaning up while in state CONNECTING due to: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:27.667 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093) for tablet 9bc0a52dc27145398e9a7b4bdd4280b5: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:27.844 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:27.844 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671),953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)[L],bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)]}]
07:56:27.846 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, attempt=12, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=750), Traces: [0ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [2ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: connection closed, [2ms] delaying RPC due to: Network error: connection closed, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [7ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [8ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [11ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [11ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [27ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [28ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [30ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [30ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [67ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [69ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [71ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [71ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [183ms] refreshing cache from master, [184ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [187ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [188ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [191ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [191ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [563ms] refreshing cache from master, [564ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [567ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [570ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [571ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [572ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [744ms] refreshing cache from master, [744ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [748ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=Deferred@1059985019(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f19483391e8@3089969 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@556426527 after retry RPC@895622009) -> (continuation of Deferred@89908544 after retry RPC@1501575241) -> (continuation of Deferred@765031220 after retry RPC@2092941877) -> (continuation of Deferred@517141542 after retry RPC@1109700985) -> (continuation of Deferred@453864486 after retry RPC@2084944199), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3a1fe13b -> wakeup thread Time-limited test -> (continuation of Deferred@556426527 after retry RPC@895622009) -> (continuation of Deferred@89908544 after retry RPC@1501575241) -> (continuation of Deferred@765031220 after retry RPC@2092941877) -> (continuation of Deferred@517141542 after retry RPC@1109700985) -> (continuation of Deferred@453864486 after retry RPC@2084944199))) after lookup
07:56:27.847 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)] connecting to peer
07:56:27.849 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:27.852 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)] cleaning up while in state CONNECTING due to: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:27.852 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093) for tablet 9bc0a52dc27145398e9a7b4bdd4280b5: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:28.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:28.408667 27285 raft_consensus.cc:493] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 953ec1e7cb1345bd986b2b0171b6bcc2)
07:56:28.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:28.408759 27286 raft_consensus.cc:493] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 953ec1e7cb1345bd986b2b0171b6bcc2)
07:56:28.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:28.409000 27286 raft_consensus.cc:515] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:28.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:28.408953 27285 raft_consensus.cc:515] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:28.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:28.410658 27285 leader_election.cc:290] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 953ec1e7cb1345bd986b2b0171b6bcc2 (127.25.235.67:36093), 3dd0f8e1dbda4e408e9caa288299ab5d (127.25.235.65:44671)
07:56:28.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:28.410745 27286 leader_election.cc:290] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 953ec1e7cb1345bd986b2b0171b6bcc2 (127.25.235.67:36093), bd0c2de15db9438f9df3c22ffa0bed6c (127.25.235.66:42989)
07:56:28.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:28.412034 26839 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.235.67:36093: connect: Connection refused (error 111)
07:56:28.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:28.426015 26839 leader_election.cc:336] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 953ec1e7cb1345bd986b2b0171b6bcc2 (127.25.235.67:36093): Network error: Client connection negotiation failed: client connection to 127.25.235.67:36093: connect: Connection refused (error 111)
07:56:28.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:28.426117 26976 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.235.67:36093: connect: Connection refused (error 111)
07:56:28.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:28.426870 26903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bc0a52dc27145398e9a7b4bdd4280b5" candidate_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" is_pre_election: true
07:56:28.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:28.427141 26903 raft_consensus.cc:2468] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd0c2de15db9438f9df3c22ffa0bed6c in term 1.
07:56:28.428 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:28.427754 26975 leader_election.cc:304] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3dd0f8e1dbda4e408e9caa288299ab5d, bd0c2de15db9438f9df3c22ffa0bed6c; no voters: 
07:56:28.428 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:28.428234 27285 raft_consensus.cc:2804] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [term 1 FOLLOWER]: Leader pre-election won for term 2
07:56:28.428 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:28.428401 27285 raft_consensus.cc:493] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 953ec1e7cb1345bd986b2b0171b6bcc2)
07:56:28.428 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:28.428526 27285 raft_consensus.cc:3060] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [term 1 FOLLOWER]: Advancing to term 2
07:56:28.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:28.428812 26976 leader_election.cc:336] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 953ec1e7cb1345bd986b2b0171b6bcc2 (127.25.235.67:36093): Network error: Client connection negotiation failed: client connection to 127.25.235.67:36093: connect: Connection refused (error 111)
07:56:28.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:28.431689 27285 raft_consensus.cc:515] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:28.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:28.432763 27285 leader_election.cc:290] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [CANDIDATE]: Term 2 election: Requested vote from peers 953ec1e7cb1345bd986b2b0171b6bcc2 (127.25.235.67:36093), 3dd0f8e1dbda4e408e9caa288299ab5d (127.25.235.65:44671)
07:56:28.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:28.433099 27041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bc0a52dc27145398e9a7b4bdd4280b5" candidate_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" is_pre_election: true
07:56:28.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:28.433094 26903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bc0a52dc27145398e9a7b4bdd4280b5" candidate_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d"
07:56:28.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:28.433308 26903 raft_consensus.cc:3060] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 1 FOLLOWER]: Advancing to term 2
07:56:28.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:28.433398 27041 raft_consensus.cc:2393] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3dd0f8e1dbda4e408e9caa288299ab5d in current term 2: Already voted for candidate bd0c2de15db9438f9df3c22ffa0bed6c in this term.
07:56:28.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:28.434149 26838 leader_election.cc:304] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3dd0f8e1dbda4e408e9caa288299ab5d; no voters: 953ec1e7cb1345bd986b2b0171b6bcc2, bd0c2de15db9438f9df3c22ffa0bed6c
07:56:28.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:28.434610 26976 leader_election.cc:336] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 953ec1e7cb1345bd986b2b0171b6bcc2 (127.25.235.67:36093): Network error: Client connection negotiation failed: client connection to 127.25.235.67:36093: connect: Connection refused (error 111)
07:56:28.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:28.436120 26903 raft_consensus.cc:2468] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd0c2de15db9438f9df3c22ffa0bed6c in term 2.
07:56:28.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:28.436470 27286 raft_consensus.cc:2749] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
07:56:28.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:28.436761 26975 leader_election.cc:304] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3dd0f8e1dbda4e408e9caa288299ab5d, bd0c2de15db9438f9df3c22ffa0bed6c; no voters: 953ec1e7cb1345bd986b2b0171b6bcc2
07:56:28.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:28.437186 27285 raft_consensus.cc:2804] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [term 2 FOLLOWER]: Leader election won for term 2
07:56:28.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:28.437619 27285 raft_consensus.cc:697] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [term 2 LEADER]: Becoming Leader. State: Replica: bd0c2de15db9438f9df3c22ffa0bed6c, State: Running, Role: LEADER
07:56:28.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:28.438652 27285 consensus_queue.cc:237] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:28.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:28.442770 26724 catalog_manager.cc:5697] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c reported cstate change: term changed from 1 to 2, leader changed from 953ec1e7cb1345bd986b2b0171b6bcc2 (127.25.235.67) to bd0c2de15db9438f9df3c22ffa0bed6c (127.25.235.66). New cstate: current_term: 2 leader_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } health_report { overall_health: HEALTHY } } }
07:56:28.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:28.873664 26903 raft_consensus.cc:1275] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 2 FOLLOWER]: Refusing update from remote peer bd0c2de15db9438f9df3c22ffa0bed6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
07:56:28.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:28.874509 27285 consensus_queue.cc:1048] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
07:56:28.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:28.882035 26976 consensus_peers.cc:597] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c -> Peer 953ec1e7cb1345bd986b2b0171b6bcc2 (127.25.235.67:36093): Couldn't send request to peer 953ec1e7cb1345bd986b2b0171b6bcc2. Status: Network error: Client connection negotiation failed: client connection to 127.25.235.67:36093: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
07:56:30.823 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:30.824 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671),953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093),bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)[L]]}]
07:56:30.826 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, attempt=14, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=3730), Traces: [0ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [2ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: connection closed, [2ms] delaying RPC due to: Network error: connection closed, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [7ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [8ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [11ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [11ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [27ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [28ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [30ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [30ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [67ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [69ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [71ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [71ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [183ms] refreshing cache from master, [184ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [187ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [188ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [191ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [191ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [563ms] refreshing cache from master, [564ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [567ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [570ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [571ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [572ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [744ms] refreshing cache from master, [744ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [748ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [752ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [757ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [757ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [3724ms] refreshing cache from master, [3724ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [3728ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=Deferred@1059985019(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f19483391e8@3089969 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@556426527 after retry RPC@895622009) -> (continuation of Deferred@89908544 after retry RPC@1501575241) -> (continuation of Deferred@765031220 after retry RPC@2092941877) -> (continuation of Deferred@517141542 after retry RPC@1109700985) -> (continuation of Deferred@453864486 after retry RPC@2084944199) -> (continuation of Deferred@1184995268 after retry RPC@1591616218), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3a1fe13b -> wakeup thread Time-limited test -> (continuation of Deferred@556426527 after retry RPC@895622009) -> (continuation of Deferred@89908544 after retry RPC@1501575241) -> (continuation of Deferred@765031220 after retry RPC@2092941877) -> (continuation of Deferred@517141542 after retry RPC@1109700985) -> (continuation of Deferred@453864486 after retry RPC@2084944199) -> (continuation of Deferred@1184995268 after retry RPC@1591616218))) after lookup
07:56:30.827 [DEBUG - kudu-nio-2] (Connection.java:748) [peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)] connecting to peer
07:56:30.828 [DEBUG - kudu-nio-2] (Connection.java:757) [peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)] Successfully connected to peer
07:56:30.847 [DEBUG - kudu-nio-2] (Negotiator.java:918) Authenticated connection [id: 0x97e4ad0e, L:/127.0.0.1:35128 - R:/127.25.235.66:42989] using TOKEN/null
07:56:30.873 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.25.235.67:36093
07:56:30.875 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

07:56:30.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:30.876091 26541 external_mini_cluster.cc:1372] Running /tmp/dist-test-task2rU0iV/build/asan/bin/kudu
07:56:30.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task2rU0iV/build/asan/bin/kudu
07:56:30.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/wal
07:56:30.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/data
07:56:30.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
07:56:30.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
07:56:30.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
07:56:30.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:56:30.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:56:30.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:56:30.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/logs
07:56:30.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/data/info.pb
07:56:30.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:56:30.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
07:56:30.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
07:56:30.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
07:56:30.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
07:56:30.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:56:30.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:56:30.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:56:30.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
07:56:30.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
07:56:30.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.235.67:36093
07:56:30.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.25.235.67
07:56:30.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.25.235.126:46873,127.25.235.125:42159,127.25.235.124:39273
07:56:30.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=43721
07:56:30.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.235.67
07:56:30.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.235.84:44497
07:56:30.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:56:30.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:56:30.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
07:56:31.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:31.216441 27325 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
07:56:31.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:31.216781 27325 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
07:56:31.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:31.216866 27325 flags.cc:432] Enabled unsafe flag: --never_fsync=true
07:56:31.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:31.225700 27325 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
07:56:31.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:31.225864 27325 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.235.67
07:56:31.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:31.236392 27325 tablet_server_runner.cc:78] Tablet server non-default flags:
07:56:31.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:56:31.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.235.84:44497
07:56:31.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:56:31.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
07:56:31.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/data
07:56:31.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/wal
07:56:31.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:56:31.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:56:31.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.235.67:36093
07:56:31.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
07:56:31.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:56:31.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:56:31.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/data/info.pb
07:56:31.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.235.67
07:56:31.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=43721
07:56:31.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.25.235.126:46873,127.25.235.125:42159,127.25.235.124:39273
07:56:31.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
07:56:31.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:56:31.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
07:56:31.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
07:56:31.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:56:31.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.25.235.67
07:56:31.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/logs
07:56:31.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:56:31.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
07:56:31.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
07:56:31.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
07:56:31.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
07:56:31.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
07:56:31.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
07:56:31.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 07:43:22 UTC on bdcb31816ec0
07:56:31.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12770
07:56:31.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
07:56:31.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:31.238076 27325 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
07:56:31.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:31.239840 27325 file_cache.cc:504] Constructed file cache file cache with capacity 419430
07:56:31.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:31.249681 27331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:31.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:31.249857 27330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:31.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:31.250520 27325 server_base.cc:1061] running on GCE node
07:56:31.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:31.250592 27333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:31.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:31.251426 27325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
07:56:31.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:31.252441 27325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
07:56:31.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:31.253674 27325 hybrid_clock.cc:648] HybridClock initialized: now 1783497391253587 us; error 74 us; skew 500 ppm
07:56:31.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:31.254040 27325 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
07:56:31.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:31.256440 27325 webserver.cc:533] Webserver started at http://127.25.235.67:43721/ using document root <none> and password file <none>
07:56:31.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:31.257313 27325 fs_manager.cc:362] Metadata directory not provided
07:56:31.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:31.257467 27325 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
07:56:31.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:31.264205 27325 fs_manager.cc:714] Time spent opening directory manager: real 0.004s	user 0.002s	sys 0.004s
07:56:31.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:31.267692 27339 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
07:56:31.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:31.269408 27325 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.002s
07:56:31.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:31.269577 27325 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/data,/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/wal
07:56:31.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "953ec1e7cb1345bd986b2b0171b6bcc2"
07:56:31.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 07:56:25 on dist-test-slave-63m1"
07:56:31.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:31.270479 27325 fs_report.cc:389] FS layout report
07:56:31.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
07:56:31.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/wal
07:56:31.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/wal
07:56:31.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/data/data
07:56:31.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
07:56:31.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
07:56:31.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
07:56:31.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
07:56:31.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
07:56:31.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
07:56:31.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
07:56:31.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
07:56:31.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
07:56:31.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
07:56:31.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
07:56:31.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:31.317140 27325 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
07:56:31.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:31.317988 27325 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
07:56:31.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:31.318249 27325 kserver.cc:163] Server-wide thread pool size limit: 3276
07:56:31.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:31.319162 27325 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
07:56:31.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:31.321887 27346 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
07:56:31.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:31.328199 27325 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
07:56:31.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:31.328326 27325 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.007s	user 0.001s	sys 0.000s
07:56:31.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:31.328431 27325 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
07:56:31.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:31.331650 27325 ts_tablet_manager.cc:616] Registered 1 tablets
07:56:31.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:31.331831 27325 ts_tablet_manager.cc:595] Time spent register tablets: real 0.003s	user 0.001s	sys 0.000s
07:56:31.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:31.332392 27346 tablet_bootstrap.cc:492] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2: Bootstrap starting.
07:56:31.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:31.359513 27346 log.cc:826] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2: Log is configured to *not* fsync() on all Append() calls
07:56:31.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:31.383389 27346 tablet_bootstrap.cc:492] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
07:56:31.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:31.384577 27346 tablet_bootstrap.cc:492] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2: Bootstrap complete.
07:56:31.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:31.386562 27346 ts_tablet_manager.cc:1403] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2: Time spent bootstrapping tablet: real 0.055s	user 0.035s	sys 0.015s
07:56:31.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:31.395221 27346 raft_consensus.cc:359] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:31.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:31.396493 27346 raft_consensus.cc:740] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 953ec1e7cb1345bd986b2b0171b6bcc2, State: Initialized, Role: FOLLOWER
07:56:31.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:31.396564 27325 rpc_server.cc:307] RPC server started. Bound to: 127.25.235.67:36093
07:56:31.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:31.396625 27453 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.235.67:36093 every 8 connection(s)
07:56:31.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:31.397508 27346 consensus_queue.cc:260] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:31.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:31.398566 27325 server_base.cc:1193] Dumped server information to /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/data/info.pb
07:56:31.407 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
07:56:31.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:31.399592 27346 ts_tablet_manager.cc:1434] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2: Time spent starting tablet: real 0.013s	user 0.013s	sys 0.000s
07:56:31.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:31.406306 26541 external_mini_cluster.cc:1434] Started /tmp/dist-test-task2rU0iV/build/asan/bin/kudu as pid 27325
07:56:31.449 [DEBUG - kudu-nio-2] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 6 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
07:56:31.450 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.25.235.66:42989
07:56:31.451 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

07:56:31.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:31.451761 26541 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-task2rU0iV/build/asan/bin/kudu pid 26957
07:56:31.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:31.462164 27456 heartbeater.cc:344] Connected to a master server at 127.25.235.126:46873
07:56:31.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:31.462548 27456 heartbeater.cc:461] Registering TS with master...
07:56:31.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:31.463779 27456 heartbeater.cc:507] Master 127.25.235.126:46873 requested a full tablet report, sending...
07:56:31.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:31.467360 26582 ts_manager.cc:194] Re-registered known tserver with Master: 953ec1e7cb1345bd986b2b0171b6bcc2 (127.25.235.67:36093)
07:56:31.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:31.475952 27455 heartbeater.cc:344] Connected to a master server at 127.25.235.124:39273
07:56:31.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:31.476250 27455 heartbeater.cc:461] Registering TS with master...
07:56:31.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:31.476835 27455 heartbeater.cc:507] Master 127.25.235.124:39273 requested a full tablet report, sending...
07:56:31.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:31.479184 26724 ts_manager.cc:194] Re-registered known tserver with Master: 953ec1e7cb1345bd986b2b0171b6bcc2 (127.25.235.67:36093)
07:56:31.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:31.481592 26724 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.235.67:43951
07:56:31.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:31.486508 27455 heartbeater.cc:499] Master 127.25.235.124:39273 was elected leader, sending a full tablet report...
07:56:31.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:31.486779 27409 raft_consensus.cc:3060] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [term 1 FOLLOWER]: Advancing to term 2
07:56:31.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:31.491492 27457 heartbeater.cc:344] Connected to a master server at 127.25.235.125:42159
07:56:31.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:31.491703 27457 heartbeater.cc:461] Registering TS with master...
07:56:31.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:31.492113 27457 heartbeater.cc:507] Master 127.25.235.125:42159 requested a full tablet report, sending...
07:56:31.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:31.492718 27081 generic_service.cc:196] Checking for leaks (request via RPC)
07:56:31.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:31.505645 26653 ts_manager.cc:194] Re-registered known tserver with Master: 953ec1e7cb1345bd986b2b0171b6bcc2 (127.25.235.67:36093)
07:56:31.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:31.603941 26541 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task2rU0iV/build/asan/bin/kudu with pid 26957
07:56:31.622 [DEBUG - kudu-nio-2] (Connection.java:239) [peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)] handling channelInactive
07:56:31.622 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
07:56:31.623 [DEBUG - kudu-nio-2] (Connection.java:709) [peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)] cleaning up while in state READY due to: connection closed
07:56:31.623 [DEBUG - Time-limited test] (Connection.java:748) [peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)] connecting to peer
07:56:31.624 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989
07:56:31.624 [DEBUG - kudu-nio-3] (Connection.java:709) [peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)] cleaning up while in state CONNECTING due to: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989
07:56:31.624 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989) for tablet 9bc0a52dc27145398e9a7b4bdd4280b5: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989
07:56:31.643 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:31.643 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671),953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093),bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)[L]]}]
07:56:31.644 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, attempt=2, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=21), Traces: [0ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [1ms] received response from server bd0c2de15db9438f9df3c22ffa0bed6c: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [1ms] delaying RPC due to: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [15ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [19ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=Deferred@2074680120(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f19483391e8@6ff4fb20 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@79078898 -> wakeup thread Time-limited test)) after lookup
07:56:31.645 [DEBUG - kudu-nio-2] (Connection.java:748) [peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)] connecting to peer
07:56:31.646 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989
07:56:31.646 [DEBUG - kudu-nio-4] (Connection.java:709) [peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)] cleaning up while in state CONNECTING due to: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989
07:56:31.646 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989) for tablet 9bc0a52dc27145398e9a7b4bdd4280b5: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989
07:56:31.662 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:31.663 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671),953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093),bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)[L]]}]
07:56:31.663 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, attempt=4, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=40), Traces: [0ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [1ms] received response from server bd0c2de15db9438f9df3c22ffa0bed6c: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [1ms] delaying RPC due to: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [15ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [19ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [21ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [23ms] received response from server bd0c2de15db9438f9df3c22ffa0bed6c: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [23ms] delaying RPC due to: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [39ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=Deferred@2074680120(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f19483391e8@6ff4fb20 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@433478646 after retry RPC@1557152540), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@79078898 -> wakeup thread Time-limited test -> (continuation of Deferred@433478646 after retry RPC@1557152540))) after lookup
07:56:31.664 [DEBUG - kudu-nio-2] (Connection.java:748) [peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)] connecting to peer
07:56:31.665 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989
07:56:31.666 [DEBUG - kudu-nio-5] (Connection.java:709) [peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)] cleaning up while in state CONNECTING due to: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989
07:56:31.666 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989) for tablet 9bc0a52dc27145398e9a7b4bdd4280b5: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989
07:56:31.702 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:31.703 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671),953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093),bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)[L]]}]
07:56:31.703 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, attempt=6, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=80), Traces: [0ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [1ms] received response from server bd0c2de15db9438f9df3c22ffa0bed6c: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [1ms] delaying RPC due to: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [15ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [19ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [21ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [23ms] received response from server bd0c2de15db9438f9df3c22ffa0bed6c: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [23ms] delaying RPC due to: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [39ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [41ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [43ms] received response from server bd0c2de15db9438f9df3c22ffa0bed6c: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [43ms] delaying RPC due to: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [76ms] refreshing cache from master, [76ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [79ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=Deferred@2074680120(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f19483391e8@6ff4fb20 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@433478646 after retry RPC@1557152540) -> (continuation of Deferred@1503822417 after retry RPC@120906458), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@79078898 -> wakeup thread Time-limited test -> (continuation of Deferred@433478646 after retry RPC@1557152540) -> (continuation of Deferred@1503822417 after retry RPC@120906458))) after lookup
07:56:31.704 [DEBUG - kudu-nio-2] (Connection.java:748) [peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)] connecting to peer
07:56:31.705 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989
07:56:31.705 [DEBUG - kudu-nio-6] (Connection.java:709) [peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)] cleaning up while in state CONNECTING due to: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989
07:56:31.705 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989) for tablet 9bc0a52dc27145398e9a7b4bdd4280b5: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989
07:56:31.783 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:31.783 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671),953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093),bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)[L]]}]
07:56:31.784 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, attempt=8, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=161), Traces: [0ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [1ms] received response from server bd0c2de15db9438f9df3c22ffa0bed6c: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [1ms] delaying RPC due to: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [15ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [19ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [21ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [23ms] received response from server bd0c2de15db9438f9df3c22ffa0bed6c: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [23ms] delaying RPC due to: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [39ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [41ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [43ms] received response from server bd0c2de15db9438f9df3c22ffa0bed6c: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [43ms] delaying RPC due to: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [76ms] refreshing cache from master, [76ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [79ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [81ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [82ms] received response from server bd0c2de15db9438f9df3c22ffa0bed6c: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [82ms] delaying RPC due to: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [156ms] refreshing cache from master, [156ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [160ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=Deferred@2074680120(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f19483391e8@6ff4fb20 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@433478646 after retry RPC@1557152540) -> (continuation of Deferred@1503822417 after retry RPC@120906458) -> (continuation of Deferred@904736472 after retry RPC@359610509), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@79078898 -> wakeup thread Time-limited test -> (continuation of Deferred@433478646 after retry RPC@1557152540) -> (continuation of Deferred@1503822417 after retry RPC@120906458) -> (continuation of Deferred@904736472 after retry RPC@359610509))) after lookup
07:56:31.785 [DEBUG - kudu-nio-2] (Connection.java:748) [peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)] connecting to peer
07:56:31.787 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989
07:56:31.787 [DEBUG - kudu-nio-7] (Connection.java:709) [peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)] cleaning up while in state CONNECTING due to: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989
07:56:31.787 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989) for tablet 9bc0a52dc27145398e9a7b4bdd4280b5: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989
07:56:31.902 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:31.902 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671),953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093),bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)[L]]}]
07:56:31.903 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, attempt=10, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=279), Traces: [0ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [1ms] received response from server bd0c2de15db9438f9df3c22ffa0bed6c: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [1ms] delaying RPC due to: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [15ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [19ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [21ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [23ms] received response from server bd0c2de15db9438f9df3c22ffa0bed6c: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [23ms] delaying RPC due to: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [39ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [41ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [43ms] received response from server bd0c2de15db9438f9df3c22ffa0bed6c: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [43ms] delaying RPC due to: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [76ms] refreshing cache from master, [76ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [79ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [81ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [82ms] received response from server bd0c2de15db9438f9df3c22ffa0bed6c: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [82ms] delaying RPC due to: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [156ms] refreshing cache from master, [156ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [160ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [162ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [164ms] received response from server bd0c2de15db9438f9df3c22ffa0bed6c: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [165ms] delaying RPC due to: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [275ms] refreshing cache from master, [276ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [279ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=Deferred@2074680120(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f19483391e8@6ff4fb20 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@433478646 after retry RPC@1557152540) -> (continuation of Deferred@1503822417 after retry RPC@120906458) -> (continuation of Deferred@904736472 after retry RPC@359610509) -> (continuation of Deferred@132077033 after retry RPC@558693133), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@79078898 -> wakeup thread Time-limited test -> (continuation of Deferred@433478646 after retry RPC@1557152540) -> (continuation of Deferred@1503822417 after retry RPC@120906458) -> (continuation of Deferred@904736472 after retry RPC@359610509) -> (continuation of Deferred@132077033 after retry RPC@558693133))) after lookup
07:56:31.904 [DEBUG - kudu-nio-2] (Connection.java:748) [peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)] connecting to peer
07:56:31.905 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989
07:56:31.905 [DEBUG - kudu-nio-0] (Connection.java:709) [peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)] cleaning up while in state CONNECTING due to: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989
07:56:31.905 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989) for tablet 9bc0a52dc27145398e9a7b4bdd4280b5: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989
07:56:32.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:32.763825 27475 raft_consensus.cc:493] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader bd0c2de15db9438f9df3c22ffa0bed6c)
07:56:32.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:32.764014 27475 raft_consensus.cc:515] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:32.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:32.764734 27475 leader_election.cc:290] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 953ec1e7cb1345bd986b2b0171b6bcc2 (127.25.235.67:36093), bd0c2de15db9438f9df3c22ffa0bed6c (127.25.235.66:42989)
07:56:32.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:32.767933 26838 leader_election.cc:336] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer bd0c2de15db9438f9df3c22ffa0bed6c (127.25.235.66:42989): Network error: Client connection negotiation failed: client connection to 127.25.235.66:42989: connect: Connection refused (error 111)
07:56:32.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:32.772090 27409 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bc0a52dc27145398e9a7b4bdd4280b5" candidate_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" is_pre_election: true
07:56:32.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:32.773270 26839 leader_election.cc:304] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3dd0f8e1dbda4e408e9caa288299ab5d; no voters: 953ec1e7cb1345bd986b2b0171b6bcc2, bd0c2de15db9438f9df3c22ffa0bed6c
07:56:32.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:32.773622 27475 raft_consensus.cc:2749] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
07:56:32.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:32.993657 27480 raft_consensus.cc:493] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader bd0c2de15db9438f9df3c22ffa0bed6c)
07:56:32.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:32.993973 27480 raft_consensus.cc:515] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:32.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:32.995537 27480 leader_election.cc:290] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3dd0f8e1dbda4e408e9caa288299ab5d (127.25.235.65:44671), bd0c2de15db9438f9df3c22ffa0bed6c (127.25.235.66:42989)
07:56:32.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:32.997689 27341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.235.66:42989: connect: Connection refused (error 111)
07:56:32.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:32.999512 27341 leader_election.cc:336] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer bd0c2de15db9438f9df3c22ffa0bed6c (127.25.235.66:42989): Network error: Client connection negotiation failed: client connection to 127.25.235.66:42989: connect: Connection refused (error 111)
07:56:33.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:33.003378 26903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bc0a52dc27145398e9a7b4bdd4280b5" candidate_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" is_pre_election: true
07:56:33.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:33.003628 26903 raft_consensus.cc:2468] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 953ec1e7cb1345bd986b2b0171b6bcc2 in term 2.
07:56:33.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:33.004176 27341 leader_election.cc:304] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3dd0f8e1dbda4e408e9caa288299ab5d, 953ec1e7cb1345bd986b2b0171b6bcc2; no voters: bd0c2de15db9438f9df3c22ffa0bed6c
07:56:33.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:33.004572 27480 raft_consensus.cc:2804] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [term 2 FOLLOWER]: Leader pre-election won for term 3
07:56:33.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:33.004716 27480 raft_consensus.cc:493] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [term 2 FOLLOWER]: Starting leader election (detected failure of leader bd0c2de15db9438f9df3c22ffa0bed6c)
07:56:33.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:33.004820 27480 raft_consensus.cc:3060] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [term 2 FOLLOWER]: Advancing to term 3
07:56:33.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:33.006907 27480 raft_consensus.cc:515] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:33.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:33.007530 27480 leader_election.cc:290] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [CANDIDATE]: Term 3 election: Requested vote from peers 3dd0f8e1dbda4e408e9caa288299ab5d (127.25.235.65:44671), bd0c2de15db9438f9df3c22ffa0bed6c (127.25.235.66:42989)
07:56:33.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:33.008133 26903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bc0a52dc27145398e9a7b4bdd4280b5" candidate_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d"
07:56:33.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:33.008342 26903 raft_consensus.cc:3060] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 2 FOLLOWER]: Advancing to term 3
07:56:33.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:33.009140 27341 leader_election.cc:336] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer bd0c2de15db9438f9df3c22ffa0bed6c (127.25.235.66:42989): Network error: Client connection negotiation failed: client connection to 127.25.235.66:42989: connect: Connection refused (error 111)
07:56:33.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:33.010548 26903 raft_consensus.cc:2468] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 953ec1e7cb1345bd986b2b0171b6bcc2 in term 3.
07:56:33.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:33.011025 27341 leader_election.cc:304] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3dd0f8e1dbda4e408e9caa288299ab5d, 953ec1e7cb1345bd986b2b0171b6bcc2; no voters: bd0c2de15db9438f9df3c22ffa0bed6c
07:56:33.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:33.011349 27480 raft_consensus.cc:2804] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [term 3 FOLLOWER]: Leader election won for term 3
07:56:33.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:33.011710 27480 raft_consensus.cc:697] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [term 3 LEADER]: Becoming Leader. State: Replica: 953ec1e7cb1345bd986b2b0171b6bcc2, State: Running, Role: LEADER
07:56:33.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:33.012288 27480 consensus_queue.cc:237] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:33.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:33.016149 26724 catalog_manager.cc:5697] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 reported cstate change: term changed from 2 to 3, leader changed from bd0c2de15db9438f9df3c22ffa0bed6c (127.25.235.66) to 953ec1e7cb1345bd986b2b0171b6bcc2 (127.25.235.67). New cstate: current_term: 3 leader_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } health_report { overall_health: UNKNOWN } } }
07:56:33.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:33.418596 26903 raft_consensus.cc:1275] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 3 FOLLOWER]: Refusing update from remote peer 953ec1e7cb1345bd986b2b0171b6bcc2: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
07:56:33.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:33.419561 27480 consensus_queue.cc:1048] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
07:56:33.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:33.426826 27341 consensus_peers.cc:597] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 -> Peer bd0c2de15db9438f9df3c22ffa0bed6c (127.25.235.66:42989): Couldn't send request to peer bd0c2de15db9438f9df3c22ffa0bed6c. Status: Network error: Client connection negotiation failed: client connection to 127.25.235.66:42989: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
07:56:33.942 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:33.943 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671),953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)[L],bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)]}]
07:56:33.943 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, attempt=12, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=2320), Traces: [0ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [1ms] received response from server bd0c2de15db9438f9df3c22ffa0bed6c: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [1ms] delaying RPC due to: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [15ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [19ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [21ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [23ms] received response from server bd0c2de15db9438f9df3c22ffa0bed6c: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [23ms] delaying RPC due to: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [39ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [41ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [43ms] received response from server bd0c2de15db9438f9df3c22ffa0bed6c: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [43ms] delaying RPC due to: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [76ms] refreshing cache from master, [76ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [79ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [81ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [82ms] received response from server bd0c2de15db9438f9df3c22ffa0bed6c: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [82ms] delaying RPC due to: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [156ms] refreshing cache from master, [156ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [160ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [162ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [164ms] received response from server bd0c2de15db9438f9df3c22ffa0bed6c: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [165ms] delaying RPC due to: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [275ms] refreshing cache from master, [276ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [279ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [280ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [282ms] received response from server bd0c2de15db9438f9df3c22ffa0bed6c: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [283ms] delaying RPC due to: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [2316ms] refreshing cache from master, [2316ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [2319ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=Deferred@2074680120(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f19483391e8@6ff4fb20 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@433478646 after retry RPC@1557152540) -> (continuation of Deferred@1503822417 after retry RPC@120906458) -> (continuation of Deferred@904736472 after retry RPC@359610509) -> (continuation of Deferred@132077033 after retry RPC@558693133) -> (continuation of Deferred@263722884 after retry RPC@1231684077), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@79078898 -> wakeup thread Time-limited test -> (continuation of Deferred@433478646 after retry RPC@1557152540) -> (continuation of Deferred@1503822417 after retry RPC@120906458) -> (continuation of Deferred@904736472 after retry RPC@359610509) -> (continuation of Deferred@132077033 after retry RPC@558693133) -> (continuation of Deferred@263722884 after retry RPC@1231684077))) after lookup
07:56:33.944 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)] connecting to peer
07:56:33.945 [DEBUG - kudu-nio-1] (Connection.java:757) [peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)] Successfully connected to peer
07:56:33.964 [DEBUG - kudu-nio-1] (Negotiator.java:918) Authenticated connection [id: 0xa1c4ff9d, L:/127.0.0.1:60700 - R:/127.25.235.67:36093] using TOKEN/null
07:56:33.989 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.25.235.66:42989
07:56:33.990 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

07:56:33.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:33.991539 26541 external_mini_cluster.cc:1372] Running /tmp/dist-test-task2rU0iV/build/asan/bin/kudu
07:56:33.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task2rU0iV/build/asan/bin/kudu
07:56:33.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-1/wal
07:56:33.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-1/data
07:56:33.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
07:56:33.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
07:56:33.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
07:56:33.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:56:33.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:56:33.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:56:33.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-1/logs
07:56:33.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-1/data/info.pb
07:56:33.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:56:33.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
07:56:33.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
07:56:33.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
07:56:33.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
07:56:33.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:56:33.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:56:33.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:56:33.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
07:56:33.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
07:56:33.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.235.66:42989
07:56:33.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.25.235.66
07:56:33.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.25.235.126:46873,127.25.235.125:42159,127.25.235.124:39273
07:56:33.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42391
07:56:33.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.235.66
07:56:33.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.235.84:44497
07:56:33.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:56:33.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:56:33.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
07:56:34.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:34.335372 27510 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
07:56:34.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:34.335692 27510 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
07:56:34.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:34.335780 27510 flags.cc:432] Enabled unsafe flag: --never_fsync=true
07:56:34.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:34.345043 27510 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
07:56:34.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:34.345316 27510 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.235.66
07:56:34.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:34.355953 27510 tablet_server_runner.cc:78] Tablet server non-default flags:
07:56:34.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:56:34.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.235.84:44497
07:56:34.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:56:34.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
07:56:34.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-1/data
07:56:34.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-1/wal
07:56:34.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:56:34.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:56:34.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.235.66:42989
07:56:34.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
07:56:34.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:56:34.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:56:34.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-1/data/info.pb
07:56:34.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.235.66
07:56:34.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42391
07:56:34.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.25.235.126:46873,127.25.235.125:42159,127.25.235.124:39273
07:56:34.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
07:56:34.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:56:34.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
07:56:34.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
07:56:34.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:56:34.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.25.235.66
07:56:34.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-1/logs
07:56:34.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:56:34.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
07:56:34.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
07:56:34.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
07:56:34.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
07:56:34.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
07:56:34.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
07:56:34.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 07:43:22 UTC on bdcb31816ec0
07:56:34.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12770
07:56:34.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
07:56:34.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:34.358386 27510 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
07:56:34.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:34.360316 27510 file_cache.cc:504] Constructed file cache file cache with capacity 419430
07:56:34.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:34.370779 27516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:34.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:34.370780 27515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:34.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:34.371464 27518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:34.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:34.371865 27510 server_base.cc:1061] running on GCE node
07:56:34.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:34.372539 27510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
07:56:34.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:34.373653 27510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
07:56:34.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:34.374883 27510 hybrid_clock.cc:648] HybridClock initialized: now 1783497394374826 us; error 60 us; skew 500 ppm
07:56:34.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:34.375185 27510 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
07:56:34.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:34.377665 27510 webserver.cc:533] Webserver started at http://127.25.235.66:42391/ using document root <none> and password file <none>
07:56:34.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:34.378468 27510 fs_manager.cc:362] Metadata directory not provided
07:56:34.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:34.378571 27510 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
07:56:34.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:34.384512 27510 fs_manager.cc:714] Time spent opening directory manager: real 0.004s	user 0.006s	sys 0.000s
07:56:34.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:34.387925 27524 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
07:56:34.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:34.389760 27510 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.001s	sys 0.001s
07:56:34.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:34.389928 27510 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-1/data,/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-1/wal
07:56:34.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "bd0c2de15db9438f9df3c22ffa0bed6c"
07:56:34.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 07:56:24 on dist-test-slave-63m1"
07:56:34.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:34.390672 27510 fs_report.cc:389] FS layout report
07:56:34.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
07:56:34.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-1/wal
07:56:34.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-1/wal
07:56:34.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-1/data/data
07:56:34.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
07:56:34.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
07:56:34.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
07:56:34.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
07:56:34.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
07:56:34.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
07:56:34.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
07:56:34.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
07:56:34.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
07:56:34.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
07:56:34.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
07:56:34.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:34.414078 27510 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
07:56:34.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:34.414973 27510 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
07:56:34.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:34.415246 27510 kserver.cc:163] Server-wide thread pool size limit: 3276
07:56:34.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:34.416267 27510 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
07:56:34.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:34.418584 27531 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
07:56:34.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:34.424847 27510 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
07:56:34.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:34.425016 27510 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.007s	user 0.001s	sys 0.000s
07:56:34.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:34.425123 27510 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
07:56:34.428 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:34.428138 27510 ts_tablet_manager.cc:616] Registered 1 tablets
07:56:34.428 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:34.428283 27510 ts_tablet_manager.cc:595] Time spent register tablets: real 0.003s	user 0.000s	sys 0.000s
07:56:34.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:34.428835 27531 tablet_bootstrap.cc:492] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c: Bootstrap starting.
07:56:34.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:34.453465 27531 log.cc:826] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c: Log is configured to *not* fsync() on all Append() calls
07:56:34.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:34.472954 27531 tablet_bootstrap.cc:492] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=6 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
07:56:34.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:34.473140 27510 rpc_server.cc:307] RPC server started. Bound to: 127.25.235.66:42989
07:56:34.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:34.473201 27638 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.235.66:42989 every 8 connection(s)
07:56:34.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:34.473799 27531 tablet_bootstrap.cc:492] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c: Bootstrap complete.
07:56:34.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:34.474808 27510 server_base.cc:1193] Dumped server information to /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-1/data/info.pb
07:56:34.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:34.475299 27531 ts_tablet_manager.cc:1403] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c: Time spent bootstrapping tablet: real 0.047s	user 0.032s	sys 0.011s
07:56:34.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:34.480470 26541 external_mini_cluster.cc:1434] Started /tmp/dist-test-task2rU0iV/build/asan/bin/kudu as pid 27510
07:56:34.481 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
07:56:34.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:34.490906 27531 raft_consensus.cc:359] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:34.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:34.492460 27531 raft_consensus.cc:740] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd0c2de15db9438f9df3c22ffa0bed6c, State: Initialized, Role: FOLLOWER
07:56:34.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:34.506444 27531 consensus_queue.cc:260] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:34.513 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 9 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
07:56:34.514 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.25.235.67:36093
07:56:34.514 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

07:56:34.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:34.515714 26541 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-task2rU0iV/build/asan/bin/kudu pid 27325
07:56:34.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:34.516983 27639 heartbeater.cc:344] Connected to a master server at 127.25.235.124:39273
07:56:34.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:34.517127 27531 ts_tablet_manager.cc:1434] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c: Time spent starting tablet: real 0.042s	user 0.014s	sys 0.005s
07:56:34.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:34.517431 27639 heartbeater.cc:461] Registering TS with master...
07:56:34.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:34.518591 27639 heartbeater.cc:507] Master 127.25.235.124:39273 requested a full tablet report, sending...
07:56:34.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:34.522189 27641 heartbeater.cc:344] Connected to a master server at 127.25.235.125:42159
07:56:34.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:34.522404 27641 heartbeater.cc:461] Registering TS with master...
07:56:34.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:34.523125 27641 heartbeater.cc:507] Master 127.25.235.125:42159 requested a full tablet report, sending...
07:56:34.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:34.523236 27640 heartbeater.cc:344] Connected to a master server at 127.25.235.126:46873
07:56:34.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:34.523366 27640 heartbeater.cc:461] Registering TS with master...
07:56:34.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:34.523880 27640 heartbeater.cc:507] Master 127.25.235.126:46873 requested a full tablet report, sending...
07:56:34.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:34.524816 26724 ts_manager.cc:194] Re-registered known tserver with Master: bd0c2de15db9438f9df3c22ffa0bed6c (127.25.235.66:42989)
07:56:34.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:34.525164 26653 ts_manager.cc:194] Re-registered known tserver with Master: bd0c2de15db9438f9df3c22ffa0bed6c (127.25.235.66:42989)
07:56:34.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:34.526397 26724 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.235.66:53303
07:56:34.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:34.527833 26582 ts_manager.cc:194] Re-registered known tserver with Master: bd0c2de15db9438f9df3c22ffa0bed6c (127.25.235.66:42989)
07:56:34.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:34.531599 27639 heartbeater.cc:499] Master 127.25.235.124:39273 was elected leader, sending a full tablet report...
07:56:34.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:34.540472 27449 generic_service.cc:196] Checking for leaks (request via RPC)
07:56:34.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:34.642809 26541 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task2rU0iV/build/asan/bin/kudu with pid 27325
07:56:34.661 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
07:56:34.662 [DEBUG - kudu-nio-1] (Connection.java:239) [peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)] handling channelInactive
07:56:34.662 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)] cleaning up while in state READY due to: connection closed
07:56:34.680 [DEBUG - pool-2-thread-1] (Connection.java:748) [peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)] connecting to peer
07:56:34.681 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:34.682 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)] cleaning up while in state CONNECTING due to: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:34.682 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093) for tablet 9bc0a52dc27145398e9a7b4bdd4280b5: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:34.702 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:34.703 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671),953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)[L],bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)]}]
07:56:34.703 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, attempt=3, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=41), Traces: [0ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [0ms] delaying RPC due to: Illegal state: connection is terminated, [17ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [20ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [20ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [37ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [40ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=Deferred@1574012432(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f19483391e8@5a9b2700 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4baa67a0 -> wakeup thread Time-limited test)) after lookup
07:56:34.704 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)] connecting to peer
07:56:34.705 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:34.705 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)] cleaning up while in state CONNECTING due to: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:34.705 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093) for tablet 9bc0a52dc27145398e9a7b4bdd4280b5: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:34.722 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:34.723 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671),953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)[L],bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)]}]
07:56:34.723 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, attempt=5, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=61), Traces: [0ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [0ms] delaying RPC due to: Illegal state: connection is terminated, [17ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [20ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [20ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [37ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [40ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [42ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [43ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [44ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [57ms] refreshing cache from master, [57ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [60ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=Deferred@1574012432(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f19483391e8@5a9b2700 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1906211440 after retry RPC@1762755716), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4baa67a0 -> wakeup thread Time-limited test -> (continuation of Deferred@1906211440 after retry RPC@1762755716))) after lookup
07:56:34.724 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)] connecting to peer
07:56:34.725 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:34.725 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)] cleaning up while in state CONNECTING due to: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:34.725 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093) for tablet 9bc0a52dc27145398e9a7b4bdd4280b5: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:34.802 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:34.803 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671),953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)[L],bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)]}]
07:56:34.803 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, attempt=7, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=141), Traces: [0ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [0ms] delaying RPC due to: Illegal state: connection is terminated, [17ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [20ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [20ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [37ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [40ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [42ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [43ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [44ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [57ms] refreshing cache from master, [57ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [60ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [62ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [63ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [63ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [137ms] refreshing cache from master, [137ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [140ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=Deferred@1574012432(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f19483391e8@5a9b2700 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1906211440 after retry RPC@1762755716) -> (continuation of Deferred@1034145319 after retry RPC@1483950259), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4baa67a0 -> wakeup thread Time-limited test -> (continuation of Deferred@1906211440 after retry RPC@1762755716) -> (continuation of Deferred@1034145319 after retry RPC@1483950259))) after lookup
07:56:34.804 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)] connecting to peer
07:56:34.805 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:34.805 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)] cleaning up while in state CONNECTING due to: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:34.806 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093) for tablet 9bc0a52dc27145398e9a7b4bdd4280b5: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:34.842 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:34.843 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671),953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)[L],bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)]}]
07:56:34.843 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, attempt=9, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=181), Traces: [0ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [0ms] delaying RPC due to: Illegal state: connection is terminated, [17ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [20ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [20ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [37ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [40ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [42ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [43ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [44ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [57ms] refreshing cache from master, [57ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [60ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [62ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [63ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [63ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [137ms] refreshing cache from master, [137ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [140ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [142ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [143ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [144ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [177ms] refreshing cache from master, [177ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [180ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=Deferred@1574012432(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f19483391e8@5a9b2700 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1906211440 after retry RPC@1762755716) -> (continuation of Deferred@1034145319 after retry RPC@1483950259) -> (continuation of Deferred@615224263 after retry RPC@1717530929), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4baa67a0 -> wakeup thread Time-limited test -> (continuation of Deferred@1906211440 after retry RPC@1762755716) -> (continuation of Deferred@1034145319 after retry RPC@1483950259) -> (continuation of Deferred@615224263 after retry RPC@1717530929))) after lookup
07:56:34.844 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)] connecting to peer
07:56:34.845 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:34.845 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)] cleaning up while in state CONNECTING due to: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:34.845 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093) for tablet 9bc0a52dc27145398e9a7b4bdd4280b5: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:35.483 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:35.483 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671),953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)[L],bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)]}]
07:56:35.484 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, attempt=11, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=821), Traces: [0ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [0ms] delaying RPC due to: Illegal state: connection is terminated, [17ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [20ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [20ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [37ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [40ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [42ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [43ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [44ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [57ms] refreshing cache from master, [57ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [60ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [62ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [63ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [63ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [137ms] refreshing cache from master, [137ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [140ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [142ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [143ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [144ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [177ms] refreshing cache from master, [177ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [180ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [181ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [183ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [183ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [817ms] refreshing cache from master, [817ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [821ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=Deferred@1574012432(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f19483391e8@5a9b2700 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1906211440 after retry RPC@1762755716) -> (continuation of Deferred@1034145319 after retry RPC@1483950259) -> (continuation of Deferred@615224263 after retry RPC@1717530929) -> (continuation of Deferred@1430385294 after retry RPC@1217054702), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4baa67a0 -> wakeup thread Time-limited test -> (continuation of Deferred@1906211440 after retry RPC@1762755716) -> (continuation of Deferred@1034145319 after retry RPC@1483950259) -> (continuation of Deferred@615224263 after retry RPC@1717530929) -> (continuation of Deferred@1430385294 after retry RPC@1217054702))) after lookup
07:56:35.485 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)] connecting to peer
07:56:35.486 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:35.486 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)] cleaning up while in state CONNECTING due to: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:35.486 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093) for tablet 9bc0a52dc27145398e9a7b4bdd4280b5: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:35.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:35.796181 27654 raft_consensus.cc:493] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
07:56:35.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:35.796476 27654 raft_consensus.cc:515] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:35.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:35.798316 27654 leader_election.cc:290] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 953ec1e7cb1345bd986b2b0171b6bcc2 (127.25.235.67:36093), 3dd0f8e1dbda4e408e9caa288299ab5d (127.25.235.65:44671)
07:56:35.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:35.799775 27527 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.235.67:36093: connect: Connection refused (error 111)
07:56:35.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:35.802536 27527 leader_election.cc:336] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 953ec1e7cb1345bd986b2b0171b6bcc2 (127.25.235.67:36093): Network error: Client connection negotiation failed: client connection to 127.25.235.67:36093: connect: Connection refused (error 111)
07:56:35.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:35.805802 26903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bc0a52dc27145398e9a7b4bdd4280b5" candidate_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" is_pre_election: true
07:56:35.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:35.806559 27526 leader_election.cc:304] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bd0c2de15db9438f9df3c22ffa0bed6c; no voters: 3dd0f8e1dbda4e408e9caa288299ab5d, 953ec1e7cb1345bd986b2b0171b6bcc2
07:56:35.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:35.806962 27654 raft_consensus.cc:2749] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
07:56:35.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:35.886315 27659 raft_consensus.cc:493] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 953ec1e7cb1345bd986b2b0171b6bcc2)
07:56:35.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:35.886579 27659 raft_consensus.cc:515] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:35.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:35.887403 27659 leader_election.cc:290] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 953ec1e7cb1345bd986b2b0171b6bcc2 (127.25.235.67:36093), bd0c2de15db9438f9df3c22ffa0bed6c (127.25.235.66:42989)
07:56:35.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:35.888804 26839 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.235.67:36093: connect: Connection refused (error 111) [suppressed 1 similar messages]
07:56:35.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:35.891870 26839 leader_election.cc:336] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 953ec1e7cb1345bd986b2b0171b6bcc2 (127.25.235.67:36093): Network error: Client connection negotiation failed: client connection to 127.25.235.67:36093: connect: Connection refused (error 111)
07:56:35.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:35.896047 27593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bc0a52dc27145398e9a7b4bdd4280b5" candidate_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" is_pre_election: true
07:56:35.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:35.896428 27593 raft_consensus.cc:2468] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3dd0f8e1dbda4e408e9caa288299ab5d in term 2.
07:56:35.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:35.897421 26838 leader_election.cc:304] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3dd0f8e1dbda4e408e9caa288299ab5d, bd0c2de15db9438f9df3c22ffa0bed6c; no voters: 953ec1e7cb1345bd986b2b0171b6bcc2
07:56:35.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:35.897722 27659 raft_consensus.cc:2804] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 3 FOLLOWER]: Leader pre-election won for term 4
07:56:35.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:35.897861 27659 raft_consensus.cc:493] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 953ec1e7cb1345bd986b2b0171b6bcc2)
07:56:35.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:35.897931 27659 raft_consensus.cc:3060] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 3 FOLLOWER]: Advancing to term 4
07:56:35.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:35.900132 27659 raft_consensus.cc:515] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:35.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:35.900705 27659 leader_election.cc:290] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [CANDIDATE]: Term 4 election: Requested vote from peers 953ec1e7cb1345bd986b2b0171b6bcc2 (127.25.235.67:36093), bd0c2de15db9438f9df3c22ffa0bed6c (127.25.235.66:42989)
07:56:35.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:35.901302 27593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bc0a52dc27145398e9a7b4bdd4280b5" candidate_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c"
07:56:35.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:35.901508 27593 raft_consensus.cc:3060] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [term 2 FOLLOWER]: Advancing to term 4
07:56:35.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:35.902283 26839 leader_election.cc:336] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 953ec1e7cb1345bd986b2b0171b6bcc2 (127.25.235.67:36093): Network error: Client connection negotiation failed: client connection to 127.25.235.67:36093: connect: Connection refused (error 111)
07:56:35.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:35.905656 27593 raft_consensus.cc:2468] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3dd0f8e1dbda4e408e9caa288299ab5d in term 4.
07:56:35.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:35.906103 26838 leader_election.cc:304] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3dd0f8e1dbda4e408e9caa288299ab5d, bd0c2de15db9438f9df3c22ffa0bed6c; no voters: 953ec1e7cb1345bd986b2b0171b6bcc2
07:56:35.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:35.906407 27659 raft_consensus.cc:2804] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 4 FOLLOWER]: Leader election won for term 4
07:56:35.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:35.906944 27659 raft_consensus.cc:697] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 4 LEADER]: Becoming Leader. State: Replica: 3dd0f8e1dbda4e408e9caa288299ab5d, State: Running, Role: LEADER
07:56:35.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:35.907794 27659 consensus_queue.cc:237] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 12, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:35.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:35.911608 26724 catalog_manager.cc:5697] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d reported cstate change: term changed from 3 to 4, leader changed from 953ec1e7cb1345bd986b2b0171b6bcc2 (127.25.235.67) to 3dd0f8e1dbda4e408e9caa288299ab5d (127.25.235.65). New cstate: current_term: 4 leader_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } health_report { overall_health: UNKNOWN } } }
07:56:36.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:36.425923 27593 raft_consensus.cc:1275] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [term 4 FOLLOWER]: Refusing update from remote peer 3dd0f8e1dbda4e408e9caa288299ab5d: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 4 index: 13. (index mismatch)
07:56:36.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:36.426847 27676 consensus_queue.cc:1048] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 8, Time since last communication: 0.000s
07:56:36.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:36.447455 26839 consensus_peers.cc:597] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d -> Peer 953ec1e7cb1345bd986b2b0171b6bcc2 (127.25.235.67:36093): Couldn't send request to peer 953ec1e7cb1345bd986b2b0171b6bcc2. Status: Network error: Client connection negotiation failed: client connection to 127.25.235.67:36093: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
07:56:37.002 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:37.002 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)[L],953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093),bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)]}]
07:56:37.002 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, attempt=13, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=2340), Traces: [0ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [0ms] delaying RPC due to: Illegal state: connection is terminated, [17ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [20ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [20ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [37ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [40ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [42ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [43ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [44ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [57ms] refreshing cache from master, [57ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [60ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [62ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [63ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [63ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [137ms] refreshing cache from master, [137ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [140ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [142ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [143ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [144ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [177ms] refreshing cache from master, [177ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [180ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [181ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [183ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [183ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [817ms] refreshing cache from master, [817ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [821ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [822ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [824ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [824ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [2337ms] refreshing cache from master, [2337ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [2339ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=Deferred@1574012432(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f19483391e8@5a9b2700 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1906211440 after retry RPC@1762755716) -> (continuation of Deferred@1034145319 after retry RPC@1483950259) -> (continuation of Deferred@615224263 after retry RPC@1717530929) -> (continuation of Deferred@1430385294 after retry RPC@1217054702) -> (continuation of Deferred@2037847162 after retry RPC@1734084877), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4baa67a0 -> wakeup thread Time-limited test -> (continuation of Deferred@1906211440 after retry RPC@1762755716) -> (continuation of Deferred@1034145319 after retry RPC@1483950259) -> (continuation of Deferred@615224263 after retry RPC@1717530929) -> (continuation of Deferred@1430385294 after retry RPC@1217054702) -> (continuation of Deferred@2037847162 after retry RPC@1734084877))) after lookup
07:56:37.003 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)] connecting to peer
07:56:37.004 [DEBUG - kudu-nio-0] (Connection.java:757) [peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)] Successfully connected to peer
07:56:37.025 [DEBUG - kudu-nio-0] (Negotiator.java:918) Authenticated connection [id: 0x70cf44b6, L:/127.0.0.1:57140 - R:/127.25.235.65:44671] using TOKEN/null
07:56:37.050 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.25.235.67:36093
07:56:37.050 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

07:56:37.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:37.051581 26541 external_mini_cluster.cc:1372] Running /tmp/dist-test-task2rU0iV/build/asan/bin/kudu
07:56:37.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task2rU0iV/build/asan/bin/kudu
07:56:37.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/wal
07:56:37.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/data
07:56:37.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
07:56:37.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
07:56:37.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
07:56:37.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:56:37.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:56:37.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:56:37.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/logs
07:56:37.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/data/info.pb
07:56:37.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:56:37.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
07:56:37.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
07:56:37.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
07:56:37.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
07:56:37.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:56:37.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:56:37.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:56:37.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
07:56:37.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
07:56:37.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.235.67:36093
07:56:37.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.25.235.67
07:56:37.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.25.235.126:46873,127.25.235.125:42159,127.25.235.124:39273
07:56:37.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=43721
07:56:37.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.235.67
07:56:37.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.235.84:44497
07:56:37.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:56:37.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:56:37.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
07:56:37.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:37.383490 27696 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
07:56:37.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:37.383833 27696 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
07:56:37.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:37.383919 27696 flags.cc:432] Enabled unsafe flag: --never_fsync=true
07:56:37.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:37.392828 27696 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
07:56:37.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:37.393060 27696 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.235.67
07:56:37.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:37.403622 27696 tablet_server_runner.cc:78] Tablet server non-default flags:
07:56:37.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:56:37.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.235.84:44497
07:56:37.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:56:37.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
07:56:37.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/data
07:56:37.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/wal
07:56:37.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:56:37.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:56:37.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.235.67:36093
07:56:37.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
07:56:37.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:56:37.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:56:37.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/data/info.pb
07:56:37.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.235.67
07:56:37.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=43721
07:56:37.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.25.235.126:46873,127.25.235.125:42159,127.25.235.124:39273
07:56:37.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
07:56:37.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:56:37.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
07:56:37.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
07:56:37.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:56:37.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.25.235.67
07:56:37.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/logs
07:56:37.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:56:37.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
07:56:37.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
07:56:37.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
07:56:37.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
07:56:37.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
07:56:37.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
07:56:37.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 07:43:22 UTC on bdcb31816ec0
07:56:37.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12770
07:56:37.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
07:56:37.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:37.405601 27696 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
07:56:37.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:37.407449 27696 file_cache.cc:504] Constructed file cache file cache with capacity 419430
07:56:37.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:37.417361 27701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:37.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:37.417321 27702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:37.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:37.417583 27704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:37.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:37.418835 27696 server_base.cc:1061] running on GCE node
07:56:37.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:37.419572 27696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
07:56:37.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:37.420689 27696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
07:56:37.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:37.421926 27696 hybrid_clock.cc:648] HybridClock initialized: now 1783497397421853 us; error 61 us; skew 500 ppm
07:56:37.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:37.422312 27696 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
07:56:37.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:37.424839 27696 webserver.cc:533] Webserver started at http://127.25.235.67:43721/ using document root <none> and password file <none>
07:56:37.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:37.425678 27696 fs_manager.cc:362] Metadata directory not provided
07:56:37.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:37.425822 27696 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
07:56:37.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:37.431509 27696 fs_manager.cc:714] Time spent opening directory manager: real 0.004s	user 0.003s	sys 0.002s
07:56:37.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:37.434640 27710 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
07:56:37.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:37.435984 27696 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
07:56:37.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:37.436162 27696 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/data,/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/wal
07:56:37.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "953ec1e7cb1345bd986b2b0171b6bcc2"
07:56:37.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 07:56:25 on dist-test-slave-63m1"
07:56:37.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:37.436894 27696 fs_report.cc:389] FS layout report
07:56:37.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
07:56:37.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/wal
07:56:37.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/wal
07:56:37.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/data/data
07:56:37.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
07:56:37.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
07:56:37.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
07:56:37.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
07:56:37.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
07:56:37.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
07:56:37.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
07:56:37.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
07:56:37.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
07:56:37.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
07:56:37.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
07:56:37.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:37.455214 27696 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
07:56:37.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:37.456156 27696 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
07:56:37.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:37.456499 27696 kserver.cc:163] Server-wide thread pool size limit: 3276
07:56:37.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:37.457573 27696 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
07:56:37.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:37.459805 27717 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
07:56:37.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:37.465647 27696 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
07:56:37.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:37.465758 27696 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.006s	user 0.001s	sys 0.000s
07:56:37.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:37.465895 27696 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
07:56:37.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:37.469305 27696 ts_tablet_manager.cc:616] Registered 1 tablets
07:56:37.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:37.469434 27696 ts_tablet_manager.cc:595] Time spent register tablets: real 0.004s	user 0.003s	sys 0.000s
07:56:37.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:37.469992 27717 tablet_bootstrap.cc:492] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2: Bootstrap starting.
07:56:37.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:37.493628 27717 log.cc:826] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2: Log is configured to *not* fsync() on all Append() calls
07:56:37.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:37.521687 27717 tablet_bootstrap.cc:492] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=9 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
07:56:37.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:37.522414 27717 tablet_bootstrap.cc:492] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2: Bootstrap complete.
07:56:37.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:37.523543 27696 rpc_server.cc:307] RPC server started. Bound to: 127.25.235.67:36093
07:56:37.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:37.523579 27826 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.235.67:36093 every 8 connection(s)
07:56:37.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:37.524124 27717 ts_tablet_manager.cc:1403] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2: Time spent bootstrapping tablet: real 0.055s	user 0.038s	sys 0.012s
07:56:37.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:37.525511 27696 server_base.cc:1193] Dumped server information to /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/data/info.pb
07:56:37.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:37.529392 26541 external_mini_cluster.cc:1434] Started /tmp/dist-test-task2rU0iV/build/asan/bin/kudu as pid 27696
07:56:37.530 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
07:56:37.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:37.534265 27717 raft_consensus.cc:359] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:37.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:37.535650 27717 raft_consensus.cc:740] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 953ec1e7cb1345bd986b2b0171b6bcc2, State: Initialized, Role: FOLLOWER
07:56:37.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:37.536629 27717 consensus_queue.cc:260] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 3.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:37.559 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 12 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
07:56:37.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:37.553521 27717 ts_tablet_manager.cc:1434] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2: Time spent starting tablet: real 0.029s	user 0.022s	sys 0.009s
07:56:37.585 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.25.235.65:44671
07:56:37.586 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

07:56:37.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:37.586488 26541 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-task2rU0iV/build/asan/bin/kudu pid 26817
07:56:37.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:37.596424 27827 heartbeater.cc:344] Connected to a master server at 127.25.235.124:39273
07:56:37.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:37.596441 27828 heartbeater.cc:344] Connected to a master server at 127.25.235.126:46873
07:56:37.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:37.596873 27828 heartbeater.cc:461] Registering TS with master...
07:56:37.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:37.596868 27827 heartbeater.cc:461] Registering TS with master...
07:56:37.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:37.598104 27827 heartbeater.cc:507] Master 127.25.235.124:39273 requested a full tablet report, sending...
07:56:37.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:37.600641 27828 heartbeater.cc:507] Master 127.25.235.126:46873 requested a full tablet report, sending...
07:56:37.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:37.601339 26724 ts_manager.cc:194] Re-registered known tserver with Master: 953ec1e7cb1345bd986b2b0171b6bcc2 (127.25.235.67:36093)
07:56:37.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:37.602151 26582 ts_manager.cc:194] Re-registered known tserver with Master: 953ec1e7cb1345bd986b2b0171b6bcc2 (127.25.235.67:36093)
07:56:37.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:37.602875 26724 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.235.67:53519
07:56:37.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:37.605973 27827 heartbeater.cc:499] Master 127.25.235.124:39273 was elected leader, sending a full tablet report...
07:56:37.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:37.606328 27829 heartbeater.cc:344] Connected to a master server at 127.25.235.125:42159
07:56:37.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:37.606519 27829 heartbeater.cc:461] Registering TS with master...
07:56:37.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:37.606832 27829 heartbeater.cc:507] Master 127.25.235.125:42159 requested a full tablet report, sending...
07:56:37.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:37.609264 26654 ts_manager.cc:194] Re-registered known tserver with Master: 953ec1e7cb1345bd986b2b0171b6bcc2 (127.25.235.67:36093)
07:56:37.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:37.617599 26944 generic_service.cc:196] Checking for leaks (request via RPC)
07:56:37.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:37.725580 26541 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task2rU0iV/build/asan/bin/kudu with pid 26817
07:56:37.745 [DEBUG - kudu-nio-0] (Connection.java:239) [peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)] handling channelInactive
07:56:37.746 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)] cleaning up while in state READY due to: connection closed
07:56:37.746 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
07:56:37.747 [DEBUG - Time-limited test] (Connection.java:748) [peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)] connecting to peer
07:56:37.748 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671
07:56:37.748 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671
07:56:37.749 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671) for tablet 9bc0a52dc27145398e9a7b4bdd4280b5: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671
07:56:37.762 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:37.762 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)[L],953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093),bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)]}]
07:56:37.763 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, attempt=2, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=16), Traces: [0ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [2ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [2ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [15ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=Deferred@765333306(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f19483391e8@2a0d6fe5 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@719d5d7e -> wakeup thread Time-limited test)) after lookup
07:56:37.763 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)] connecting to peer
07:56:37.764 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671
07:56:37.764 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671
07:56:37.765 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671) for tablet 9bc0a52dc27145398e9a7b4bdd4280b5: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671
07:56:37.782 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:37.782 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)[L],953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093),bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)]}]
07:56:37.782 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, attempt=4, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=35), Traces: [0ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [2ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [2ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [15ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [16ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [17ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [18ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [32ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [35ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=Deferred@765333306(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f19483391e8@2a0d6fe5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@337772332 after retry RPC@1315057347), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@719d5d7e -> wakeup thread Time-limited test -> (continuation of Deferred@337772332 after retry RPC@1315057347))) after lookup
07:56:37.783 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)] connecting to peer
07:56:37.784 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671
07:56:37.784 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671
07:56:37.784 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671) for tablet 9bc0a52dc27145398e9a7b4bdd4280b5: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671
07:56:37.801 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:37.802 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)[L],953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093),bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)]}]
07:56:37.802 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, attempt=6, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=55), Traces: [0ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [2ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [2ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [15ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [16ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [17ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [18ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [32ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [35ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [36ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [37ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [37ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [51ms] refreshing cache from master, [51ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [54ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=Deferred@765333306(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f19483391e8@2a0d6fe5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@337772332 after retry RPC@1315057347) -> (continuation of Deferred@549348424 after retry RPC@2138391210), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@719d5d7e -> wakeup thread Time-limited test -> (continuation of Deferred@337772332 after retry RPC@1315057347) -> (continuation of Deferred@549348424 after retry RPC@2138391210))) after lookup
07:56:37.803 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)] connecting to peer
07:56:37.804 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671
07:56:37.804 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671
07:56:37.804 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671) for tablet 9bc0a52dc27145398e9a7b4bdd4280b5: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671
07:56:37.902 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:37.902 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)[L],953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093),bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)]}]
07:56:37.902 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, attempt=8, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=155), Traces: [0ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [2ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [2ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [15ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [16ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [17ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [18ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [32ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [35ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [36ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [37ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [37ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [51ms] refreshing cache from master, [51ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [54ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [56ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [57ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [57ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [151ms] refreshing cache from master, [152ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [154ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=Deferred@765333306(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f19483391e8@2a0d6fe5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@337772332 after retry RPC@1315057347) -> (continuation of Deferred@549348424 after retry RPC@2138391210) -> (continuation of Deferred@1218128404 after retry RPC@497959359), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@719d5d7e -> wakeup thread Time-limited test -> (continuation of Deferred@337772332 after retry RPC@1315057347) -> (continuation of Deferred@549348424 after retry RPC@2138391210) -> (continuation of Deferred@1218128404 after retry RPC@497959359))) after lookup
07:56:37.903 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)] connecting to peer
07:56:37.904 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671
07:56:37.904 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671
07:56:37.905 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671) for tablet 9bc0a52dc27145398e9a7b4bdd4280b5: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671
07:56:38.122 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:38.122 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)[L],953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093),bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)]}]
07:56:38.123 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, attempt=10, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=375), Traces: [0ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [2ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [2ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [15ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [16ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [17ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [18ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [32ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [35ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [36ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [37ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [37ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [51ms] refreshing cache from master, [51ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [54ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [56ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [57ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [57ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [151ms] refreshing cache from master, [152ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [154ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [156ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [157ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [158ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [371ms] refreshing cache from master, [372ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [374ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=Deferred@765333306(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f19483391e8@2a0d6fe5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@337772332 after retry RPC@1315057347) -> (continuation of Deferred@549348424 after retry RPC@2138391210) -> (continuation of Deferred@1218128404 after retry RPC@497959359) -> (continuation of Deferred@5419599 after retry RPC@1864280131), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@719d5d7e -> wakeup thread Time-limited test -> (continuation of Deferred@337772332 after retry RPC@1315057347) -> (continuation of Deferred@549348424 after retry RPC@2138391210) -> (continuation of Deferred@1218128404 after retry RPC@497959359) -> (continuation of Deferred@5419599 after retry RPC@1864280131))) after lookup
07:56:38.124 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)] connecting to peer
07:56:38.125 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671
07:56:38.125 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671
07:56:38.125 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671) for tablet 9bc0a52dc27145398e9a7b4bdd4280b5: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671
07:56:38.242 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:38.243 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)[L],953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093),bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)]}]
07:56:38.243 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, attempt=12, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=496), Traces: [0ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [2ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [2ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [15ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [16ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [17ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [18ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [32ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [35ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [36ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [37ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [37ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [51ms] refreshing cache from master, [51ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [54ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [56ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [57ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [57ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [151ms] refreshing cache from master, [152ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [154ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [156ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [157ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [158ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [371ms] refreshing cache from master, [372ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [374ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [376ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [378ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [378ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [492ms] refreshing cache from master, [492ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [495ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=Deferred@765333306(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f19483391e8@2a0d6fe5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@337772332 after retry RPC@1315057347) -> (continuation of Deferred@549348424 after retry RPC@2138391210) -> (continuation of Deferred@1218128404 after retry RPC@497959359) -> (continuation of Deferred@5419599 after retry RPC@1864280131) -> (continuation of Deferred@1531685913 after retry RPC@1091394435), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@719d5d7e -> wakeup thread Time-limited test -> (continuation of Deferred@337772332 after retry RPC@1315057347) -> (continuation of Deferred@549348424 after retry RPC@2138391210) -> (continuation of Deferred@1218128404 after retry RPC@497959359) -> (continuation of Deferred@5419599 after retry RPC@1864280131) -> (continuation of Deferred@1531685913 after retry RPC@1091394435))) after lookup
07:56:38.244 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)] connecting to peer
07:56:38.245 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671
07:56:38.245 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671
07:56:38.245 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671) for tablet 9bc0a52dc27145398e9a7b4bdd4280b5: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671
07:56:38.642 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:38.643 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)[L],953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093),bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)]}]
07:56:38.643 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, attempt=14, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=896), Traces: [0ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [2ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [2ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [15ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [16ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [17ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [18ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [32ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [35ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [36ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [37ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [37ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [51ms] refreshing cache from master, [51ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [54ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [56ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [57ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [57ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [151ms] refreshing cache from master, [152ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [154ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [156ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [157ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [158ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [371ms] refreshing cache from master, [372ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [374ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [376ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [378ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [378ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [492ms] refreshing cache from master, [492ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [495ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [497ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [498ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [498ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [892ms] refreshing cache from master, [892ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [895ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=Deferred@765333306(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f19483391e8@2a0d6fe5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@337772332 after retry RPC@1315057347) -> (continuation of Deferred@549348424 after retry RPC@2138391210) -> (continuation of Deferred@1218128404 after retry RPC@497959359) -> (continuation of Deferred@5419599 after retry RPC@1864280131) -> (continuation of Deferred@1531685913 after retry RPC@1091394435) -> (continuation of Deferred@511760265 after retry RPC@905598617), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@719d5d7e -> wakeup thread Time-limited test -> (continuation of Deferred@337772332 after retry RPC@1315057347) -> (continuation of Deferred@549348424 after retry RPC@2138391210) -> (continuation of Deferred@1218128404 after retry RPC@497959359) -> (continuation of Deferred@5419599 after retry RPC@1864280131) -> (continuation of Deferred@1531685913 after retry RPC@1091394435) -> (continuation of Deferred@511760265 after retry RPC@905598617))) after lookup
07:56:38.644 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)] connecting to peer
07:56:38.645 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671
07:56:38.645 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671
07:56:38.645 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671) for tablet 9bc0a52dc27145398e9a7b4bdd4280b5: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671
07:56:38.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:38.965407 27842 raft_consensus.cc:493] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
07:56:38.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:38.965790 27842 raft_consensus.cc:515] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:38.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:38.967832 27842 leader_election.cc:290] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3dd0f8e1dbda4e408e9caa288299ab5d (127.25.235.65:44671), bd0c2de15db9438f9df3c22ffa0bed6c (127.25.235.66:42989)
07:56:38.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:38.969357 27712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.235.65:44671: connect: Connection refused (error 111)
07:56:38.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:38.971271 27712 leader_election.cc:336] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 3dd0f8e1dbda4e408e9caa288299ab5d (127.25.235.65:44671): Network error: Client connection negotiation failed: client connection to 127.25.235.65:44671: connect: Connection refused (error 111)
07:56:38.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:38.972023 27847 raft_consensus.cc:493] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3dd0f8e1dbda4e408e9caa288299ab5d)
07:56:38.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:38.972232 27847 raft_consensus.cc:515] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:38.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:38.973076 27847 leader_election.cc:290] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 953ec1e7cb1345bd986b2b0171b6bcc2 (127.25.235.67:36093), 3dd0f8e1dbda4e408e9caa288299ab5d (127.25.235.65:44671)
07:56:38.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:38.977674 27526 leader_election.cc:336] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 3dd0f8e1dbda4e408e9caa288299ab5d (127.25.235.65:44671): Network error: Client connection negotiation failed: client connection to 127.25.235.65:44671: connect: Connection refused (error 111)
07:56:38.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:38.978304 27593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bc0a52dc27145398e9a7b4bdd4280b5" candidate_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" is_pre_election: true
07:56:38.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:38.978682 27593 raft_consensus.cc:2393] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 953ec1e7cb1345bd986b2b0171b6bcc2 in current term 4: Already voted for candidate 3dd0f8e1dbda4e408e9caa288299ab5d in this term.
07:56:38.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:38.979612 27712 leader_election.cc:304] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 953ec1e7cb1345bd986b2b0171b6bcc2; no voters: 3dd0f8e1dbda4e408e9caa288299ab5d, bd0c2de15db9438f9df3c22ffa0bed6c
07:56:38.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:38.980408 27842 raft_consensus.cc:3060] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [term 3 FOLLOWER]: Advancing to term 4
07:56:38.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:38.982936 27782 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bc0a52dc27145398e9a7b4bdd4280b5" candidate_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" is_pre_election: true
07:56:38.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:38.985057 27842 raft_consensus.cc:2749] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
07:56:38.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:38.985415 27782 raft_consensus.cc:2468] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd0c2de15db9438f9df3c22ffa0bed6c in term 4.
07:56:38.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:38.987027 27527 leader_election.cc:304] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 953ec1e7cb1345bd986b2b0171b6bcc2, bd0c2de15db9438f9df3c22ffa0bed6c; no voters: 3dd0f8e1dbda4e408e9caa288299ab5d
07:56:38.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:38.987391 27847 raft_consensus.cc:2804] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [term 4 FOLLOWER]: Leader pre-election won for term 5
07:56:38.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:38.987504 27847 raft_consensus.cc:493] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3dd0f8e1dbda4e408e9caa288299ab5d)
07:56:38.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:38.987571 27847 raft_consensus.cc:3060] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [term 4 FOLLOWER]: Advancing to term 5
07:56:38.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:38.989847 27847 raft_consensus.cc:515] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:38.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:38.990417 27847 leader_election.cc:290] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [CANDIDATE]: Term 5 election: Requested vote from peers 953ec1e7cb1345bd986b2b0171b6bcc2 (127.25.235.67:36093), 3dd0f8e1dbda4e408e9caa288299ab5d (127.25.235.65:44671)
07:56:38.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:38.990944 27782 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bc0a52dc27145398e9a7b4bdd4280b5" candidate_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2"
07:56:38.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:38.991176 27782 raft_consensus.cc:3060] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [term 4 FOLLOWER]: Advancing to term 5
07:56:38.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:38.991977 27526 leader_election.cc:336] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 3dd0f8e1dbda4e408e9caa288299ab5d (127.25.235.65:44671): Network error: Client connection negotiation failed: client connection to 127.25.235.65:44671: connect: Connection refused (error 111)
07:56:38.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:38.993654 27782 raft_consensus.cc:2468] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd0c2de15db9438f9df3c22ffa0bed6c in term 5.
07:56:38.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:38.994122 27527 leader_election.cc:304] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 953ec1e7cb1345bd986b2b0171b6bcc2, bd0c2de15db9438f9df3c22ffa0bed6c; no voters: 3dd0f8e1dbda4e408e9caa288299ab5d
07:56:38.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:38.994475 27847 raft_consensus.cc:2804] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [term 5 FOLLOWER]: Leader election won for term 5
07:56:38.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:38.994915 27847 raft_consensus.cc:697] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [term 5 LEADER]: Becoming Leader. State: Replica: bd0c2de15db9438f9df3c22ffa0bed6c, State: Running, Role: LEADER
07:56:38.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:38.995456 27847 consensus_queue.cc:237] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 4.16, Last appended by leader: 16, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:39.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:39.000020 26724 catalog_manager.cc:5697] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c reported cstate change: term changed from 4 to 5, leader changed from 3dd0f8e1dbda4e408e9caa288299ab5d (127.25.235.65) to bd0c2de15db9438f9df3c22ffa0bed6c (127.25.235.66). New cstate: current_term: 5 leader_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } health_report { overall_health: HEALTHY } } }
07:56:39.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:39.460587 27526 consensus_peers.cc:597] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c -> Peer 3dd0f8e1dbda4e408e9caa288299ab5d (127.25.235.65:44671): Couldn't send request to peer 3dd0f8e1dbda4e408e9caa288299ab5d. Status: Network error: Client connection negotiation failed: client connection to 127.25.235.65:44671: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
07:56:39.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:39.572618 27782 raft_consensus.cc:1275] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [term 5 FOLLOWER]: Refusing update from remote peer bd0c2de15db9438f9df3c22ffa0bed6c: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 5 index: 17. (index mismatch)
07:56:39.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:39.573700 27847 consensus_queue.cc:1048] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 12, Time since last communication: 0.000s
07:56:39.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:39.582536 27866 mvcc.cc:204] Tried to move back new op lower bound from 7305205338230673408 to 7305205333641977856. Current Snapshot: MvccSnapshot[applied={T|T < 7305205338230673408}]
07:56:40.843 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:40.843 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671),953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093),bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)[L]]}]
07:56:40.843 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, attempt=16, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=3096), Traces: [0ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [2ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [2ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [15ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [16ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [17ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [18ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [32ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [35ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [36ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [37ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [37ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [51ms] refreshing cache from master, [51ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [54ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [56ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [57ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [57ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [151ms] refreshing cache from master, [152ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [154ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [156ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [157ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [158ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [371ms] refreshing cache from master, [372ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [374ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [376ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [378ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [378ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [492ms] refreshing cache from master, [492ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [495ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [497ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [498ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [498ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [892ms] refreshing cache from master, [892ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [895ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [897ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [898ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [898ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [3092ms] refreshing cache from master, [3092ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [3095ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=Deferred@765333306(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f19483391e8@2a0d6fe5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@337772332 after retry RPC@1315057347) -> (continuation of Deferred@549348424 after retry RPC@2138391210) -> (continuation of Deferred@1218128404 after retry RPC@497959359) -> (continuation of Deferred@5419599 after retry RPC@1864280131) -> (continuation of Deferred@1531685913 after retry RPC@1091394435) -> (continuation of Deferred@511760265 after retry RPC@905598617) -> (continuation of Deferred@1878560321 after retry RPC@139543481), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@719d5d7e -> wakeup thread Time-limited test -> (continuation of Deferred@337772332 after retry RPC@1315057347) -> (continuation of Deferred@549348424 after retry RPC@2138391210) -> (continuation of Deferred@1218128404 after retry RPC@497959359) -> (continuation of Deferred@5419599 after retry RPC@1864280131) -> (continuation of Deferred@1531685913 after retry RPC@1091394435) -> (continuation of Deferred@511760265 after retry RPC@905598617) -> (continuation of Deferred@1878560321 after retry RPC@139543481))) after lookup
07:56:40.844 [DEBUG - kudu-nio-2] (Connection.java:748) [peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)] connecting to peer
07:56:40.846 [DEBUG - kudu-nio-1] (Connection.java:757) [peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)] Successfully connected to peer
07:56:40.866 [DEBUG - kudu-nio-1] (Negotiator.java:918) Authenticated connection [id: 0x9867ffc4, L:/127.0.0.1:33330 - R:/127.25.235.66:42989] using TOKEN/null
07:56:40.894 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.25.235.65:44671
07:56:40.895 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

07:56:40.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:40.895658 26541 external_mini_cluster.cc:1372] Running /tmp/dist-test-task2rU0iV/build/asan/bin/kudu
07:56:40.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task2rU0iV/build/asan/bin/kudu
07:56:40.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-0/wal
07:56:40.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-0/data
07:56:40.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
07:56:40.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
07:56:40.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
07:56:40.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:56:40.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:56:40.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:56:40.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-0/logs
07:56:40.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-0/data/info.pb
07:56:40.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:56:40.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
07:56:40.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
07:56:40.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
07:56:40.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
07:56:40.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:56:40.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:56:40.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:56:40.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
07:56:40.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
07:56:40.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.235.65:44671
07:56:40.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.25.235.65
07:56:40.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.25.235.126:46873,127.25.235.125:42159,127.25.235.124:39273
07:56:40.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42151
07:56:40.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.235.65
07:56:40.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.235.84:44497
07:56:40.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:56:40.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:56:40.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
07:56:41.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:41.083719 27526 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.235.65:44671: connect: Connection refused (error 111) [suppressed 5 similar messages]
07:56:41.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:41.232630 27885 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
07:56:41.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:41.233001 27885 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
07:56:41.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:41.233091 27885 flags.cc:432] Enabled unsafe flag: --never_fsync=true
07:56:41.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:41.241534 27885 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
07:56:41.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:41.241781 27885 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.235.65
07:56:41.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:41.252130 27885 tablet_server_runner.cc:78] Tablet server non-default flags:
07:56:41.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:56:41.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.235.84:44497
07:56:41.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:56:41.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
07:56:41.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-0/data
07:56:41.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-0/wal
07:56:41.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:56:41.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:56:41.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.235.65:44671
07:56:41.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
07:56:41.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:56:41.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:56:41.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-0/data/info.pb
07:56:41.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.235.65
07:56:41.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42151
07:56:41.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.25.235.126:46873,127.25.235.125:42159,127.25.235.124:39273
07:56:41.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
07:56:41.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:56:41.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
07:56:41.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
07:56:41.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:56:41.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.25.235.65
07:56:41.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-0/logs
07:56:41.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:56:41.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
07:56:41.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
07:56:41.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
07:56:41.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
07:56:41.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
07:56:41.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
07:56:41.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 07:43:22 UTC on bdcb31816ec0
07:56:41.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12770
07:56:41.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
07:56:41.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:41.253787 27885 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
07:56:41.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:41.255584 27885 file_cache.cc:504] Constructed file cache file cache with capacity 419430
07:56:41.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:41.264797 27892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:41.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:41.265106 27893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:41.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:41.268239 27895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:41.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:41.268657 27885 server_base.cc:1061] running on GCE node
07:56:41.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:41.269328 27885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
07:56:41.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:41.270254 27885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
07:56:41.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:41.271515 27885 hybrid_clock.cc:648] HybridClock initialized: now 1783497401271462 us; error 42 us; skew 500 ppm
07:56:41.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:41.271904 27885 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
07:56:41.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:41.274430 27885 webserver.cc:533] Webserver started at http://127.25.235.65:42151/ using document root <none> and password file <none>
07:56:41.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:41.275240 27885 fs_manager.cc:362] Metadata directory not provided
07:56:41.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:41.275401 27885 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
07:56:41.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:41.281134 27885 fs_manager.cc:714] Time spent opening directory manager: real 0.004s	user 0.003s	sys 0.001s
07:56:41.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:41.284235 27901 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
07:56:41.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:41.285621 27885 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
07:56:41.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:41.285805 27885 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-0/data,/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-0/wal
07:56:41.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "3dd0f8e1dbda4e408e9caa288299ab5d"
07:56:41.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 07:56:24 on dist-test-slave-63m1"
07:56:41.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:41.286558 27885 fs_report.cc:389] FS layout report
07:56:41.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
07:56:41.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-0/wal
07:56:41.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-0/wal
07:56:41.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-0/data/data
07:56:41.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
07:56:41.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
07:56:41.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
07:56:41.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
07:56:41.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
07:56:41.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
07:56:41.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
07:56:41.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
07:56:41.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
07:56:41.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
07:56:41.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
07:56:41.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:41.313306 27885 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
07:56:41.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:41.314312 27885 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
07:56:41.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:41.314663 27885 kserver.cc:163] Server-wide thread pool size limit: 3276
07:56:41.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:41.315786 27885 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
07:56:41.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:41.318073 27908 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
07:56:41.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:41.324826 27885 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
07:56:41.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:41.325018 27885 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.007s	user 0.001s	sys 0.000s
07:56:41.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:41.325165 27885 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
07:56:41.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:41.328410 27885 ts_tablet_manager.cc:616] Registered 1 tablets
07:56:41.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:41.328545 27885 ts_tablet_manager.cc:595] Time spent register tablets: real 0.003s	user 0.000s	sys 0.000s
07:56:41.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:41.328974 27908 tablet_bootstrap.cc:492] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d: Bootstrap starting.
07:56:41.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:41.354058 27908 log.cc:826] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d: Log is configured to *not* fsync() on all Append() calls
07:56:41.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:41.378620 27885 rpc_server.cc:307] RPC server started. Bound to: 127.25.235.65:44671
07:56:41.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:41.378583 28015 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.235.65:44671 every 8 connection(s)
07:56:41.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:41.381294 27885 server_base.cc:1193] Dumped server information to /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-0/data/info.pb
07:56:41.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:41.384441 26541 external_mini_cluster.cc:1434] Started /tmp/dist-test-task2rU0iV/build/asan/bin/kudu as pid 27885
07:56:41.385 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
07:56:41.410 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 15 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
07:56:41.414 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.25.235.66:42989
07:56:41.415 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

07:56:41.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:41.415971 26541 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-task2rU0iV/build/asan/bin/kudu pid 27510
07:56:41.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:41.414579 27908 tablet_bootstrap.cc:492] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d: Bootstrap replayed 1/1 log segments. Stats: ops{read=16 overwritten=0 applied=16 ignored=0} inserts{seen=12 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
07:56:41.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:41.418175 27908 tablet_bootstrap.cc:492] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d: Bootstrap complete.
07:56:41.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:41.420539 27908 ts_tablet_manager.cc:1403] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d: Time spent bootstrapping tablet: real 0.092s	user 0.055s	sys 0.014s
07:56:41.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:41.438642 27908 raft_consensus.cc:359] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:41.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:41.440300 27908 raft_consensus.cc:740] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dd0f8e1dbda4e408e9caa288299ab5d, State: Initialized, Role: FOLLOWER
07:56:41.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:41.441445 27908 consensus_queue.cc:260] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 4.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:41.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:41.448132 28017 heartbeater.cc:344] Connected to a master server at 127.25.235.126:46873
07:56:41.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:41.448518 28017 heartbeater.cc:461] Registering TS with master...
07:56:41.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:41.449870 28017 heartbeater.cc:507] Master 127.25.235.126:46873 requested a full tablet report, sending...
07:56:41.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:41.453578 26582 ts_manager.cc:194] Re-registered known tserver with Master: 3dd0f8e1dbda4e408e9caa288299ab5d (127.25.235.65:44671)
07:56:41.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:41.458966 28016 heartbeater.cc:344] Connected to a master server at 127.25.235.124:39273
07:56:41.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:41.459100 28016 heartbeater.cc:461] Registering TS with master...
07:56:41.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:41.459653 28016 heartbeater.cc:507] Master 127.25.235.124:39273 requested a full tablet report, sending...
07:56:41.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:41.461211 26724 ts_manager.cc:194] Re-registered known tserver with Master: 3dd0f8e1dbda4e408e9caa288299ab5d (127.25.235.65:44671)
07:56:41.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:41.462374 26724 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.235.65:37115
07:56:41.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:41.469833 27634 generic_service.cc:196] Checking for leaks (request via RPC)
07:56:41.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:41.470781 28016 heartbeater.cc:499] Master 127.25.235.124:39273 was elected leader, sending a full tablet report...
07:56:41.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:41.471434 27908 ts_tablet_manager.cc:1434] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d: Time spent starting tablet: real 0.051s	user 0.019s	sys 0.023s
07:56:41.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:41.478966 28018 heartbeater.cc:344] Connected to a master server at 127.25.235.125:42159
07:56:41.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:41.479180 28018 heartbeater.cc:461] Registering TS with master...
07:56:41.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:41.479570 28018 heartbeater.cc:507] Master 127.25.235.125:42159 requested a full tablet report, sending...
07:56:41.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:41.480976 26654 ts_manager.cc:194] Re-registered known tserver with Master: 3dd0f8e1dbda4e408e9caa288299ab5d (127.25.235.65:44671)
07:56:41.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:41.593024 26541 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task2rU0iV/build/asan/bin/kudu with pid 27510
07:56:41.612 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
07:56:41.612 [DEBUG - kudu-nio-1] (Connection.java:239) [peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)] handling channelInactive
07:56:41.612 [DEBUG - kudu-nio-1] (Connection.java:709) [peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)] cleaning up while in state READY due to: connection closed
07:56:41.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:41.613222 28031 negotiation.cc:336] Failed RPC negotiation. Trace:
07:56:41.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 07:56:41.580721 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.25.235.66:53097 (local address 127.25.235.65:44671)
07:56:41.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 07:56:41.591829 (+ 11108us) server_negotiation.cc:207] Beginning negotiation
07:56:41.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 07:56:41.591840 (+    11us) server_negotiation.cc:400] Waiting for connection header
07:56:41.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 07:56:41.591887 (+    47us) server_negotiation.cc:408] Connection header received
07:56:41.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 07:56:41.592014 (+   127us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
07:56:41.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 07:56:41.592023 (+     9us) server_negotiation.cc:462] Received NEGOTIATE request from client
07:56:41.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 07:56:41.592352 (+   329us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
07:56:41.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 07:56:41.592644 (+   292us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
07:56:41.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 07:56:41.611533 (+ 18889us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.235.66:53097: BlockingRecv error: recv got EOF from 127.25.235.66:53097 (error 108)
07:56:41.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":10926,"thread_start_us":10766,"threads_started":1}
07:56:41.619 [DEBUG - pool-2-thread-1] (Connection.java:748) [peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)] connecting to peer
07:56:41.621 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989
07:56:41.621 [DEBUG - kudu-nio-2] (Connection.java:709) [peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)] cleaning up while in state CONNECTING due to: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989
07:56:41.621 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989) for tablet 9bc0a52dc27145398e9a7b4bdd4280b5: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989
07:56:41.642 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:41.643 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671),953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093),bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)[L]]}]
07:56:41.643 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, attempt=3, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=30), Traces: [0ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [0ms] delaying RPC due to: Illegal state: connection is terminated, [7ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [9ms] received response from server bd0c2de15db9438f9df3c22ffa0bed6c: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [9ms] delaying RPC due to: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [30ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=Deferred@1141243786(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f19483391e8@48c4100a -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@67aac703 -> wakeup thread Time-limited test)) after lookup
07:56:41.643 [DEBUG - kudu-nio-2] (Connection.java:748) [peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)] connecting to peer
07:56:41.644 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989
07:56:41.644 [DEBUG - kudu-nio-3] (Connection.java:709) [peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)] cleaning up while in state CONNECTING due to: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989
07:56:41.645 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989) for tablet 9bc0a52dc27145398e9a7b4bdd4280b5: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989
07:56:41.662 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:41.662 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671),953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093),bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)[L]]}]
07:56:41.663 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, attempt=5, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=50), Traces: [0ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [0ms] delaying RPC due to: Illegal state: connection is terminated, [7ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [9ms] received response from server bd0c2de15db9438f9df3c22ffa0bed6c: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [9ms] delaying RPC due to: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [30ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [31ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [32ms] received response from server bd0c2de15db9438f9df3c22ffa0bed6c: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [33ms] delaying RPC due to: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [47ms] refreshing cache from master, [47ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [50ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=Deferred@1141243786(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f19483391e8@48c4100a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@219395724 after retry RPC@2010919566), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@67aac703 -> wakeup thread Time-limited test -> (continuation of Deferred@219395724 after retry RPC@2010919566))) after lookup
07:56:41.664 [DEBUG - kudu-nio-2] (Connection.java:748) [peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)] connecting to peer
07:56:41.665 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989
07:56:41.665 [DEBUG - kudu-nio-4] (Connection.java:709) [peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)] cleaning up while in state CONNECTING due to: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989
07:56:41.665 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989) for tablet 9bc0a52dc27145398e9a7b4bdd4280b5: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989
07:56:41.722 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:41.723 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671),953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093),bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)[L]]}]
07:56:41.723 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, attempt=7, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=111), Traces: [0ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [0ms] delaying RPC due to: Illegal state: connection is terminated, [7ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [9ms] received response from server bd0c2de15db9438f9df3c22ffa0bed6c: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [9ms] delaying RPC due to: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [30ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [31ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [32ms] received response from server bd0c2de15db9438f9df3c22ffa0bed6c: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [33ms] delaying RPC due to: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [47ms] refreshing cache from master, [47ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [50ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [51ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [53ms] received response from server bd0c2de15db9438f9df3c22ffa0bed6c: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [53ms] delaying RPC due to: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [107ms] refreshing cache from master, [107ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [110ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=Deferred@1141243786(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f19483391e8@48c4100a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@219395724 after retry RPC@2010919566) -> (continuation of Deferred@1278350874 after retry RPC@1525421392), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@67aac703 -> wakeup thread Time-limited test -> (continuation of Deferred@219395724 after retry RPC@2010919566) -> (continuation of Deferred@1278350874 after retry RPC@1525421392))) after lookup
07:56:41.724 [DEBUG - kudu-nio-2] (Connection.java:748) [peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)] connecting to peer
07:56:41.725 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989
07:56:41.725 [DEBUG - kudu-nio-5] (Connection.java:709) [peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)] cleaning up while in state CONNECTING due to: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989
07:56:41.725 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989) for tablet 9bc0a52dc27145398e9a7b4bdd4280b5: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989
07:56:41.823 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:41.823 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671),953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093),bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)[L]]}]
07:56:41.823 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, attempt=9, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=211), Traces: [0ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [0ms] delaying RPC due to: Illegal state: connection is terminated, [7ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [9ms] received response from server bd0c2de15db9438f9df3c22ffa0bed6c: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [9ms] delaying RPC due to: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [30ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [31ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [32ms] received response from server bd0c2de15db9438f9df3c22ffa0bed6c: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [33ms] delaying RPC due to: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [47ms] refreshing cache from master, [47ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [50ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [51ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [53ms] received response from server bd0c2de15db9438f9df3c22ffa0bed6c: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [53ms] delaying RPC due to: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [107ms] refreshing cache from master, [107ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [110ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [112ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [113ms] received response from server bd0c2de15db9438f9df3c22ffa0bed6c: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [113ms] delaying RPC due to: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [207ms] refreshing cache from master, [207ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [210ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=Deferred@1141243786(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f19483391e8@48c4100a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@219395724 after retry RPC@2010919566) -> (continuation of Deferred@1278350874 after retry RPC@1525421392) -> (continuation of Deferred@1686200238 after retry RPC@1585283323), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@67aac703 -> wakeup thread Time-limited test -> (continuation of Deferred@219395724 after retry RPC@2010919566) -> (continuation of Deferred@1278350874 after retry RPC@1525421392) -> (continuation of Deferred@1686200238 after retry RPC@1585283323))) after lookup
07:56:41.824 [DEBUG - kudu-nio-2] (Connection.java:748) [peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)] connecting to peer
07:56:41.825 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989
07:56:41.825 [DEBUG - kudu-nio-6] (Connection.java:709) [peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)] cleaning up while in state CONNECTING due to: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989
07:56:41.826 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989) for tablet 9bc0a52dc27145398e9a7b4bdd4280b5: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989
07:56:42.782 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:42.782 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671),953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093),bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)[L]]}]
07:56:42.783 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, attempt=11, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=1170), Traces: [0ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [0ms] delaying RPC due to: Illegal state: connection is terminated, [7ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [9ms] received response from server bd0c2de15db9438f9df3c22ffa0bed6c: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [9ms] delaying RPC due to: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [30ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [31ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [32ms] received response from server bd0c2de15db9438f9df3c22ffa0bed6c: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [33ms] delaying RPC due to: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [47ms] refreshing cache from master, [47ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [50ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [51ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [53ms] received response from server bd0c2de15db9438f9df3c22ffa0bed6c: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [53ms] delaying RPC due to: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [107ms] refreshing cache from master, [107ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [110ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [112ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [113ms] received response from server bd0c2de15db9438f9df3c22ffa0bed6c: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [113ms] delaying RPC due to: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [207ms] refreshing cache from master, [207ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [210ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [212ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [214ms] received response from server bd0c2de15db9438f9df3c22ffa0bed6c: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [214ms] delaying RPC due to: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [1166ms] refreshing cache from master, [1167ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [1170ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=Deferred@1141243786(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f19483391e8@48c4100a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@219395724 after retry RPC@2010919566) -> (continuation of Deferred@1278350874 after retry RPC@1525421392) -> (continuation of Deferred@1686200238 after retry RPC@1585283323) -> (continuation of Deferred@1738169537 after retry RPC@1439855249), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@67aac703 -> wakeup thread Time-limited test -> (continuation of Deferred@219395724 after retry RPC@2010919566) -> (continuation of Deferred@1278350874 after retry RPC@1525421392) -> (continuation of Deferred@1686200238 after retry RPC@1585283323) -> (continuation of Deferred@1738169537 after retry RPC@1439855249))) after lookup
07:56:42.783 [DEBUG - kudu-nio-2] (Connection.java:748) [peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)] connecting to peer
07:56:42.785 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989
07:56:42.785 [DEBUG - kudu-nio-7] (Connection.java:709) [peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)] cleaning up while in state CONNECTING due to: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989
07:56:42.785 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989) for tablet 9bc0a52dc27145398e9a7b4bdd4280b5: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989
07:56:42.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:42.827458 28032 raft_consensus.cc:493] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
07:56:42.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:42.827702 28032 raft_consensus.cc:515] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:42.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:42.829421 28032 leader_election.cc:290] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 953ec1e7cb1345bd986b2b0171b6bcc2 (127.25.235.67:36093), bd0c2de15db9438f9df3c22ffa0bed6c (127.25.235.66:42989)
07:56:42.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:42.830577 27903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.235.66:42989: connect: Connection refused (error 111)
07:56:42.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:42.832326 27903 leader_election.cc:336] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer bd0c2de15db9438f9df3c22ffa0bed6c (127.25.235.66:42989): Network error: Client connection negotiation failed: client connection to 127.25.235.66:42989: connect: Connection refused (error 111)
07:56:42.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:42.837385 27782 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bc0a52dc27145398e9a7b4bdd4280b5" candidate_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" is_pre_election: true
07:56:42.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:42.838346 27904 leader_election.cc:304] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3dd0f8e1dbda4e408e9caa288299ab5d; no voters: 953ec1e7cb1345bd986b2b0171b6bcc2, bd0c2de15db9438f9df3c22ffa0bed6c
07:56:42.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:42.838792 28032 raft_consensus.cc:2749] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
07:56:42.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:42.865748 28037 raft_consensus.cc:493] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader bd0c2de15db9438f9df3c22ffa0bed6c)
07:56:42.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:42.866063 28037 raft_consensus.cc:515] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:42.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:42.867049 28037 leader_election.cc:290] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3dd0f8e1dbda4e408e9caa288299ab5d (127.25.235.65:44671), bd0c2de15db9438f9df3c22ffa0bed6c (127.25.235.66:42989)
07:56:42.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:42.871110 27712 leader_election.cc:336] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer bd0c2de15db9438f9df3c22ffa0bed6c (127.25.235.66:42989): Network error: Client connection negotiation failed: client connection to 127.25.235.66:42989: connect: Connection refused (error 111)
07:56:42.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:42.876232 27971 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bc0a52dc27145398e9a7b4bdd4280b5" candidate_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" candidate_term: 6 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" is_pre_election: true
07:56:42.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:42.876619 27971 raft_consensus.cc:2468] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 953ec1e7cb1345bd986b2b0171b6bcc2 in term 4.
07:56:42.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:42.877705 27712 leader_election.cc:304] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3dd0f8e1dbda4e408e9caa288299ab5d, 953ec1e7cb1345bd986b2b0171b6bcc2; no voters: bd0c2de15db9438f9df3c22ffa0bed6c
07:56:42.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:42.878039 28037 raft_consensus.cc:2804] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [term 5 FOLLOWER]: Leader pre-election won for term 6
07:56:42.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:42.878170 28037 raft_consensus.cc:493] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [term 5 FOLLOWER]: Starting leader election (detected failure of leader bd0c2de15db9438f9df3c22ffa0bed6c)
07:56:42.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:42.878247 28037 raft_consensus.cc:3060] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [term 5 FOLLOWER]: Advancing to term 6
07:56:42.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:42.880455 28037 raft_consensus.cc:515] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:42.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:42.881307 28037 leader_election.cc:290] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [CANDIDATE]: Term 6 election: Requested vote from peers 3dd0f8e1dbda4e408e9caa288299ab5d (127.25.235.65:44671), bd0c2de15db9438f9df3c22ffa0bed6c (127.25.235.66:42989)
07:56:42.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:42.881829 27971 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bc0a52dc27145398e9a7b4bdd4280b5" candidate_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" candidate_term: 6 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d"
07:56:42.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:42.882057 27971 raft_consensus.cc:3060] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 4 FOLLOWER]: Advancing to term 6
07:56:42.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:42.882987 27712 leader_election.cc:336] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer bd0c2de15db9438f9df3c22ffa0bed6c (127.25.235.66:42989): Network error: Client connection negotiation failed: client connection to 127.25.235.66:42989: connect: Connection refused (error 111)
07:56:42.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:42.886409 27971 raft_consensus.cc:2468] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 953ec1e7cb1345bd986b2b0171b6bcc2 in term 6.
07:56:42.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:42.886916 27712 leader_election.cc:304] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3dd0f8e1dbda4e408e9caa288299ab5d, 953ec1e7cb1345bd986b2b0171b6bcc2; no voters: bd0c2de15db9438f9df3c22ffa0bed6c
07:56:42.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:42.887328 28037 raft_consensus.cc:2804] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [term 6 FOLLOWER]: Leader election won for term 6
07:56:42.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:42.888001 28037 raft_consensus.cc:697] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [term 6 LEADER]: Becoming Leader. State: Replica: 953ec1e7cb1345bd986b2b0171b6bcc2, State: Running, Role: LEADER
07:56:42.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:42.888783 28037 consensus_queue.cc:237] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 5.20, Last appended by leader: 20, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:42.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:42.892716 26724 catalog_manager.cc:5697] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 reported cstate change: term changed from 5 to 6, leader changed from bd0c2de15db9438f9df3c22ffa0bed6c (127.25.235.66) to 953ec1e7cb1345bd986b2b0171b6bcc2 (127.25.235.67). New cstate: current_term: 6 leader_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } health_report { overall_health: UNKNOWN } } }
07:56:43.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.360411 27971 raft_consensus.cc:1275] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 6 FOLLOWER]: Refusing update from remote peer 953ec1e7cb1345bd986b2b0171b6bcc2: Log matching property violated. Preceding OpId in replica: term: 4 index: 16. Preceding OpId from leader: term: 6 index: 21. (index mismatch)
07:56:43.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.361413 28037 consensus_queue.cc:1048] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 16, Time since last communication: 0.000s
07:56:43.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:43.375517 27712 consensus_peers.cc:597] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 -> Peer bd0c2de15db9438f9df3c22ffa0bed6c (127.25.235.66:42989): Couldn't send request to peer bd0c2de15db9438f9df3c22ffa0bed6c. Status: Network error: Client connection negotiation failed: client connection to 127.25.235.66:42989: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
07:56:44.182 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:44.183 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671),953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)[L],bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)]}]
07:56:44.183 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, attempt=13, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=2571), Traces: [0ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [0ms] delaying RPC due to: Illegal state: connection is terminated, [7ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [9ms] received response from server bd0c2de15db9438f9df3c22ffa0bed6c: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [9ms] delaying RPC due to: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [30ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [31ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [32ms] received response from server bd0c2de15db9438f9df3c22ffa0bed6c: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [33ms] delaying RPC due to: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [47ms] refreshing cache from master, [47ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [50ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [51ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [53ms] received response from server bd0c2de15db9438f9df3c22ffa0bed6c: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [53ms] delaying RPC due to: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [107ms] refreshing cache from master, [107ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [110ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [112ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [113ms] received response from server bd0c2de15db9438f9df3c22ffa0bed6c: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [113ms] delaying RPC due to: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [207ms] refreshing cache from master, [207ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [210ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [212ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [214ms] received response from server bd0c2de15db9438f9df3c22ffa0bed6c: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [214ms] delaying RPC due to: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [1166ms] refreshing cache from master, [1167ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [1170ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [1171ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [1173ms] received response from server bd0c2de15db9438f9df3c22ffa0bed6c: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [1173ms] delaying RPC due to: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [2567ms] refreshing cache from master, [2567ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [2570ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=Deferred@1141243786(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f19483391e8@48c4100a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@219395724 after retry RPC@2010919566) -> (continuation of Deferred@1278350874 after retry RPC@1525421392) -> (continuation of Deferred@1686200238 after retry RPC@1585283323) -> (continuation of Deferred@1738169537 after retry RPC@1439855249) -> (continuation of Deferred@1204339810 after retry RPC@21524720), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@67aac703 -> wakeup thread Time-limited test -> (continuation of Deferred@219395724 after retry RPC@2010919566) -> (continuation of Deferred@1278350874 after retry RPC@1525421392) -> (continuation of Deferred@1686200238 after retry RPC@1585283323) -> (continuation of Deferred@1738169537 after retry RPC@1439855249) -> (continuation of Deferred@1204339810 after retry RPC@21524720))) after lookup
07:56:44.184 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)] connecting to peer
07:56:44.186 [DEBUG - kudu-nio-0] (Connection.java:757) [peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)] Successfully connected to peer
07:56:44.206 [DEBUG - kudu-nio-0] (Negotiator.java:918) Authenticated connection [id: 0x42b4a708, L:/127.0.0.1:57936 - R:/127.25.235.67:36093] using TOKEN/null
07:56:44.235 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.25.235.66:42989
07:56:44.237 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

07:56:44.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.238000 26541 external_mini_cluster.cc:1372] Running /tmp/dist-test-task2rU0iV/build/asan/bin/kudu
07:56:44.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task2rU0iV/build/asan/bin/kudu
07:56:44.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-1/wal
07:56:44.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-1/data
07:56:44.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
07:56:44.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
07:56:44.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
07:56:44.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:56:44.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:56:44.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:56:44.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-1/logs
07:56:44.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-1/data/info.pb
07:56:44.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:56:44.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
07:56:44.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
07:56:44.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
07:56:44.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
07:56:44.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:56:44.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:56:44.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:56:44.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
07:56:44.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
07:56:44.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.235.66:42989
07:56:44.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.25.235.66
07:56:44.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.25.235.126:46873,127.25.235.125:42159,127.25.235.124:39273
07:56:44.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42391
07:56:44.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.235.66
07:56:44.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.235.84:44497
07:56:44.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:56:44.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:56:44.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
07:56:44.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:44.252998 27712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.235.66:42989: connect: Connection refused (error 111) [suppressed 4 similar messages]
07:56:44.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:44.635038 28068 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
07:56:44.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:44.635370 28068 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
07:56:44.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:44.635464 28068 flags.cc:432] Enabled unsafe flag: --never_fsync=true
07:56:44.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:44.644929 28068 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
07:56:44.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:44.645109 28068 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.235.66
07:56:44.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.655321 28068 tablet_server_runner.cc:78] Tablet server non-default flags:
07:56:44.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:56:44.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.235.84:44497
07:56:44.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:56:44.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
07:56:44.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-1/data
07:56:44.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-1/wal
07:56:44.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:56:44.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:56:44.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.235.66:42989
07:56:44.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
07:56:44.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:56:44.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:56:44.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-1/data/info.pb
07:56:44.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.235.66
07:56:44.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42391
07:56:44.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.25.235.126:46873,127.25.235.125:42159,127.25.235.124:39273
07:56:44.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
07:56:44.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:56:44.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
07:56:44.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
07:56:44.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:56:44.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.25.235.66
07:56:44.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-1/logs
07:56:44.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:56:44.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
07:56:44.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
07:56:44.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
07:56:44.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
07:56:44.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
07:56:44.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
07:56:44.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 07:43:22 UTC on bdcb31816ec0
07:56:44.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12770
07:56:44.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
07:56:44.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.656996 28068 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
07:56:44.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.658831 28068 file_cache.cc:504] Constructed file cache file cache with capacity 419430
07:56:44.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:44.669335 28076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:44.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:44.669400 28074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:44.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:44.669307 28073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:44.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.671633 28068 server_base.cc:1061] running on GCE node
07:56:44.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.672264 28068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
07:56:44.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.673244 28068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
07:56:44.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.674435 28068 hybrid_clock.cc:648] HybridClock initialized: now 1783497404674363 us; error 77 us; skew 500 ppm
07:56:44.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.674743 28068 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
07:56:44.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.676999 28068 webserver.cc:533] Webserver started at http://127.25.235.66:42391/ using document root <none> and password file <none>
07:56:44.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.677783 28068 fs_manager.cc:362] Metadata directory not provided
07:56:44.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.677881 28068 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
07:56:44.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.684340 28068 fs_manager.cc:714] Time spent opening directory manager: real 0.004s	user 0.006s	sys 0.000s
07:56:44.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.687326 28082 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
07:56:44.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.688858 28068 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.002s
07:56:44.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.689061 28068 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-1/data,/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-1/wal
07:56:44.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "bd0c2de15db9438f9df3c22ffa0bed6c"
07:56:44.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 07:56:24 on dist-test-slave-63m1"
07:56:44.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.689743 28068 fs_report.cc:389] FS layout report
07:56:44.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
07:56:44.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-1/wal
07:56:44.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-1/wal
07:56:44.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-1/data/data
07:56:44.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
07:56:44.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
07:56:44.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
07:56:44.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
07:56:44.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
07:56:44.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
07:56:44.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
07:56:44.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
07:56:44.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
07:56:44.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
07:56:44.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
07:56:44.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.706157 28068 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
07:56:44.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.707115 28068 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
07:56:44.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.707367 28068 kserver.cc:163] Server-wide thread pool size limit: 3276
07:56:44.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.708302 28068 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
07:56:44.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.710433 28089 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
07:56:44.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.717641 28068 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
07:56:44.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.717734 28068 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.008s	user 0.001s	sys 0.000s
07:56:44.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.717842 28068 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
07:56:44.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.721082 28068 ts_tablet_manager.cc:616] Registered 1 tablets
07:56:44.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.721174 28068 ts_tablet_manager.cc:595] Time spent register tablets: real 0.003s	user 0.003s	sys 0.000s
07:56:44.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.721714 28089 tablet_bootstrap.cc:492] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c: Bootstrap starting.
07:56:44.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.749356 28089 log.cc:826] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c: Log is configured to *not* fsync() on all Append() calls
07:56:44.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.769407 28068 rpc_server.cc:307] RPC server started. Bound to: 127.25.235.66:42989
07:56:44.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.769477 28196 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.235.66:42989 every 8 connection(s)
07:56:44.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.770882 28068 server_base.cc:1193] Dumped server information to /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-1/data/info.pb
07:56:44.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.779887 26541 external_mini_cluster.cc:1434] Started /tmp/dist-test-task2rU0iV/build/asan/bin/kudu as pid 28068
07:56:44.781 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
07:56:44.794 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 18 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
07:56:44.795 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.25.235.67:36093
07:56:44.795 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

07:56:44.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.796170 26541 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-task2rU0iV/build/asan/bin/kudu pid 27696
07:56:44.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.827770 28089 tablet_bootstrap.cc:492] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c: Bootstrap replayed 1/1 log segments. Stats: ops{read=20 overwritten=0 applied=20 ignored=0} inserts{seen=15 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
07:56:44.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.828722 28089 tablet_bootstrap.cc:492] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c: Bootstrap complete.
07:56:44.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.842593 28089 ts_tablet_manager.cc:1403] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c: Time spent bootstrapping tablet: real 0.121s	user 0.048s	sys 0.018s
07:56:44.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.847638 27822 generic_service.cc:196] Checking for leaks (request via RPC)
07:56:44.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.852869 28198 heartbeater.cc:344] Connected to a master server at 127.25.235.126:46873
07:56:44.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.853320 28198 heartbeater.cc:461] Registering TS with master...
07:56:44.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.854596 28198 heartbeater.cc:507] Master 127.25.235.126:46873 requested a full tablet report, sending...
07:56:44.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.857458 28089 raft_consensus.cc:359] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:44.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.858218 26582 ts_manager.cc:194] Re-registered known tserver with Master: bd0c2de15db9438f9df3c22ffa0bed6c (127.25.235.66:42989)
07:56:44.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.858353 28089 raft_consensus.cc:740] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd0c2de15db9438f9df3c22ffa0bed6c, State: Initialized, Role: FOLLOWER
07:56:44.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.859127 28089 consensus_queue.cc:260] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 20, Last appended: 5.20, Last appended by leader: 20, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:44.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.861095 28197 heartbeater.cc:344] Connected to a master server at 127.25.235.124:39273
07:56:44.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.861291 28197 heartbeater.cc:461] Registering TS with master...
07:56:44.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.861961 28197 heartbeater.cc:507] Master 127.25.235.124:39273 requested a full tablet report, sending...
07:56:44.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.863957 28089 ts_tablet_manager.cc:1434] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c: Time spent starting tablet: real 0.021s	user 0.008s	sys 0.002s
07:56:44.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.865674 26724 ts_manager.cc:194] Re-registered known tserver with Master: bd0c2de15db9438f9df3c22ffa0bed6c (127.25.235.66:42989)
07:56:44.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.867542 26724 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.235.66:42577
07:56:44.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.868547 28199 heartbeater.cc:344] Connected to a master server at 127.25.235.125:42159
07:56:44.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.868712 28199 heartbeater.cc:461] Registering TS with master...
07:56:44.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.869323 28199 heartbeater.cc:507] Master 127.25.235.125:42159 requested a full tablet report, sending...
07:56:44.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.871021 26654 ts_manager.cc:194] Re-registered known tserver with Master: bd0c2de15db9438f9df3c22ffa0bed6c (127.25.235.66:42989)
07:56:44.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.871721 28197 heartbeater.cc:499] Master 127.25.235.124:39273 was elected leader, sending a full tablet report...
07:56:44.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.962071 26541 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task2rU0iV/build/asan/bin/kudu with pid 27696
07:56:44.980 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
07:56:44.981 [DEBUG - kudu-nio-0] (Connection.java:239) [peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)] handling channelInactive
07:56:44.981 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)] cleaning up while in state READY due to: connection closed
07:56:44.999 [DEBUG - pool-2-thread-1] (Connection.java:748) [peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)] connecting to peer
07:56:45.000 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:45.001 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)] cleaning up while in state CONNECTING due to: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:45.001 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093) for tablet 9bc0a52dc27145398e9a7b4bdd4280b5: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:45.022 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:45.022 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671),953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)[L],bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)]}]
07:56:45.023 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, attempt=3, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=42), Traces: [0ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [0ms] delaying RPC due to: Illegal state: connection is terminated, [18ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [20ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [21ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [41ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=Deferred@1520354337(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f19483391e8@1893f072 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@57172aa5 -> wakeup thread Time-limited test)) after lookup
07:56:45.023 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)] connecting to peer
07:56:45.024 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:45.024 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)] cleaning up while in state CONNECTING due to: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:45.024 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093) for tablet 9bc0a52dc27145398e9a7b4bdd4280b5: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:45.042 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:45.042 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671),953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)[L],bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)]}]
07:56:45.043 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, attempt=5, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=62), Traces: [0ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [0ms] delaying RPC due to: Illegal state: connection is terminated, [18ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [20ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [21ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [41ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [42ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [43ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [43ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [58ms] refreshing cache from master, [58ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [61ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=Deferred@1520354337(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f19483391e8@1893f072 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@35172308 after retry RPC@1503658322), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@57172aa5 -> wakeup thread Time-limited test -> (continuation of Deferred@35172308 after retry RPC@1503658322))) after lookup
07:56:45.043 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)] connecting to peer
07:56:45.044 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:45.045 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)] cleaning up while in state CONNECTING due to: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:45.045 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093) for tablet 9bc0a52dc27145398e9a7b4bdd4280b5: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:45.062 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:45.062 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671),953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)[L],bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)]}]
07:56:45.063 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, attempt=7, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=81), Traces: [0ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [0ms] delaying RPC due to: Illegal state: connection is terminated, [18ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [20ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [21ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [41ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [42ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [43ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [43ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [58ms] refreshing cache from master, [58ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [61ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [62ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [64ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [64ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [77ms] refreshing cache from master, [78ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [81ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=Deferred@1520354337(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f19483391e8@1893f072 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@35172308 after retry RPC@1503658322) -> (continuation of Deferred@312780575 after retry RPC@1272422024), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@57172aa5 -> wakeup thread Time-limited test -> (continuation of Deferred@35172308 after retry RPC@1503658322) -> (continuation of Deferred@312780575 after retry RPC@1272422024))) after lookup
07:56:45.064 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)] connecting to peer
07:56:45.065 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:45.065 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)] cleaning up while in state CONNECTING due to: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:45.065 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093) for tablet 9bc0a52dc27145398e9a7b4bdd4280b5: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:45.162 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:45.162 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671),953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)[L],bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)]}]
07:56:45.162 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, attempt=9, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=181), Traces: [0ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [0ms] delaying RPC due to: Illegal state: connection is terminated, [18ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [20ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [21ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [41ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [42ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [43ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [43ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [58ms] refreshing cache from master, [58ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [61ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [62ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [64ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [64ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [77ms] refreshing cache from master, [78ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [81ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [82ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [84ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [84ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [178ms] refreshing cache from master, [178ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [181ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=Deferred@1520354337(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f19483391e8@1893f072 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@35172308 after retry RPC@1503658322) -> (continuation of Deferred@312780575 after retry RPC@1272422024) -> (continuation of Deferred@460080381 after retry RPC@1815657158), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@57172aa5 -> wakeup thread Time-limited test -> (continuation of Deferred@35172308 after retry RPC@1503658322) -> (continuation of Deferred@312780575 after retry RPC@1272422024) -> (continuation of Deferred@460080381 after retry RPC@1815657158))) after lookup
07:56:45.163 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)] connecting to peer
07:56:45.164 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:45.164 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)] cleaning up while in state CONNECTING due to: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:45.164 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093) for tablet 9bc0a52dc27145398e9a7b4bdd4280b5: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:46.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:46.134953 28212 raft_consensus.cc:493] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 953ec1e7cb1345bd986b2b0171b6bcc2)
07:56:46.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:46.135140 28212 raft_consensus.cc:515] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:46.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:46.135799 28212 leader_election.cc:290] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 953ec1e7cb1345bd986b2b0171b6bcc2 (127.25.235.67:36093), bd0c2de15db9438f9df3c22ffa0bed6c (127.25.235.66:42989)
07:56:46.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:46.138288 27904 leader_election.cc:336] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 953ec1e7cb1345bd986b2b0171b6bcc2 (127.25.235.67:36093): Network error: Client connection negotiation failed: client connection to 127.25.235.67:36093: connect: Connection refused (error 111)
07:56:46.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:46.144035 28151 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bc0a52dc27145398e9a7b4bdd4280b5" candidate_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" candidate_term: 7 candidate_status { last_received { term: 6 index: 24 } } ignore_live_leader: false dest_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" is_pre_election: true
07:56:46.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:46.144472 28151 raft_consensus.cc:2468] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3dd0f8e1dbda4e408e9caa288299ab5d in term 5.
07:56:46.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:46.145462 27903 leader_election.cc:304] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3dd0f8e1dbda4e408e9caa288299ab5d, bd0c2de15db9438f9df3c22ffa0bed6c; no voters: 953ec1e7cb1345bd986b2b0171b6bcc2
07:56:46.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:46.145754 28212 raft_consensus.cc:2804] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 6 FOLLOWER]: Leader pre-election won for term 7
07:56:46.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:46.145875 28212 raft_consensus.cc:493] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 6 FOLLOWER]: Starting leader election (detected failure of leader 953ec1e7cb1345bd986b2b0171b6bcc2)
07:56:46.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:46.145947 28212 raft_consensus.cc:3060] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 6 FOLLOWER]: Advancing to term 7
07:56:46.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:46.147974 28212 raft_consensus.cc:515] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:46.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:46.148638 28212 leader_election.cc:290] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [CANDIDATE]: Term 7 election: Requested vote from peers 953ec1e7cb1345bd986b2b0171b6bcc2 (127.25.235.67:36093), bd0c2de15db9438f9df3c22ffa0bed6c (127.25.235.66:42989)
07:56:46.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:46.149325 28151 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bc0a52dc27145398e9a7b4bdd4280b5" candidate_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" candidate_term: 7 candidate_status { last_received { term: 6 index: 24 } } ignore_live_leader: false dest_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c"
07:56:46.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:46.149559 28151 raft_consensus.cc:3060] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [term 5 FOLLOWER]: Advancing to term 7
07:56:46.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:46.150236 27904 leader_election.cc:336] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 953ec1e7cb1345bd986b2b0171b6bcc2 (127.25.235.67:36093): Network error: Client connection negotiation failed: client connection to 127.25.235.67:36093: connect: Connection refused (error 111)
07:56:46.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:46.153683 28151 raft_consensus.cc:2468] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3dd0f8e1dbda4e408e9caa288299ab5d in term 7.
07:56:46.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:46.154176 27903 leader_election.cc:304] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3dd0f8e1dbda4e408e9caa288299ab5d, bd0c2de15db9438f9df3c22ffa0bed6c; no voters: 953ec1e7cb1345bd986b2b0171b6bcc2
07:56:46.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:46.154553 28212 raft_consensus.cc:2804] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 7 FOLLOWER]: Leader election won for term 7
07:56:46.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:46.154994 28212 raft_consensus.cc:697] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 7 LEADER]: Becoming Leader. State: Replica: 3dd0f8e1dbda4e408e9caa288299ab5d, State: Running, Role: LEADER
07:56:46.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:46.155586 28212 consensus_queue.cc:237] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 6.24, Last appended by leader: 24, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:46.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:46.159562 26724 catalog_manager.cc:5697] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d reported cstate change: term changed from 6 to 7, leader changed from 953ec1e7cb1345bd986b2b0171b6bcc2 (127.25.235.67) to 3dd0f8e1dbda4e408e9caa288299ab5d (127.25.235.65). New cstate: current_term: 7 leader_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } health_report { overall_health: UNKNOWN } } }
07:56:46.163 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:46.164 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671),953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)[L],bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)]}]
07:56:46.164 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, attempt=11, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=1183), Traces: [0ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [0ms] delaying RPC due to: Illegal state: connection is terminated, [18ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [20ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [21ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [41ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [42ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [43ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [43ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [58ms] refreshing cache from master, [58ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [61ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [62ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [64ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [64ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [77ms] refreshing cache from master, [78ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [81ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [82ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [84ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [84ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [178ms] refreshing cache from master, [178ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [181ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [182ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [183ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [183ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [1178ms] refreshing cache from master, [1178ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [1182ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=Deferred@1520354337(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f19483391e8@1893f072 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@35172308 after retry RPC@1503658322) -> (continuation of Deferred@312780575 after retry RPC@1272422024) -> (continuation of Deferred@460080381 after retry RPC@1815657158) -> (continuation of Deferred@237949464 after retry RPC@585556770), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@57172aa5 -> wakeup thread Time-limited test -> (continuation of Deferred@35172308 after retry RPC@1503658322) -> (continuation of Deferred@312780575 after retry RPC@1272422024) -> (continuation of Deferred@460080381 after retry RPC@1815657158) -> (continuation of Deferred@237949464 after retry RPC@585556770))) after lookup
07:56:46.165 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)] connecting to peer
07:56:46.166 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:46.166 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)] cleaning up while in state CONNECTING due to: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:46.166 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093) for tablet 9bc0a52dc27145398e9a7b4bdd4280b5: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:46.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:46.568372 27904 consensus_peers.cc:597] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d -> Peer 953ec1e7cb1345bd986b2b0171b6bcc2 (127.25.235.67:36093): Couldn't send request to peer 953ec1e7cb1345bd986b2b0171b6bcc2. Status: Network error: Client connection negotiation failed: client connection to 127.25.235.67:36093: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
07:56:46.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:46.775820 28151 raft_consensus.cc:1275] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [term 7 FOLLOWER]: Refusing update from remote peer 3dd0f8e1dbda4e408e9caa288299ab5d: Log matching property violated. Preceding OpId in replica: term: 5 index: 20. Preceding OpId from leader: term: 7 index: 25. (index mismatch)
07:56:46.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:46.776836 28212 consensus_queue.cc:1048] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 20, Time since last communication: 0.000s
07:56:47.722 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:47.722 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)[L],953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093),bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)]}]
07:56:47.723 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, attempt=13, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=2741), Traces: [0ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [0ms] delaying RPC due to: Illegal state: connection is terminated, [18ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [20ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [21ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [41ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [42ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [43ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [43ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [58ms] refreshing cache from master, [58ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [61ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [62ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [64ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [64ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [77ms] refreshing cache from master, [78ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [81ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [82ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [84ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [84ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [178ms] refreshing cache from master, [178ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [181ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [182ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [183ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [183ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [1178ms] refreshing cache from master, [1178ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [1182ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [1184ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [1185ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [1185ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [2738ms] refreshing cache from master, [2738ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [2741ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=Deferred@1520354337(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f19483391e8@1893f072 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@35172308 after retry RPC@1503658322) -> (continuation of Deferred@312780575 after retry RPC@1272422024) -> (continuation of Deferred@460080381 after retry RPC@1815657158) -> (continuation of Deferred@237949464 after retry RPC@585556770) -> (continuation of Deferred@1186080623 after retry RPC@1406673768), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@57172aa5 -> wakeup thread Time-limited test -> (continuation of Deferred@35172308 after retry RPC@1503658322) -> (continuation of Deferred@312780575 after retry RPC@1272422024) -> (continuation of Deferred@460080381 after retry RPC@1815657158) -> (continuation of Deferred@237949464 after retry RPC@585556770) -> (continuation of Deferred@1186080623 after retry RPC@1406673768))) after lookup
07:56:47.723 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)] connecting to peer
07:56:47.724 [DEBUG - kudu-nio-7] (Connection.java:757) [peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)] Successfully connected to peer
07:56:47.750 [DEBUG - kudu-nio-7] (Negotiator.java:918) Authenticated connection [id: 0x73865c47, L:/127.0.0.1:55384 - R:/127.25.235.65:44671] using TOKEN/null
07:56:47.773 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.25.235.67:36093
07:56:47.773 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

07:56:47.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:47.774684 26541 external_mini_cluster.cc:1372] Running /tmp/dist-test-task2rU0iV/build/asan/bin/kudu
07:56:47.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task2rU0iV/build/asan/bin/kudu
07:56:47.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/wal
07:56:47.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/data
07:56:47.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
07:56:47.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
07:56:47.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
07:56:47.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:56:47.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:56:47.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:56:47.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/logs
07:56:47.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/data/info.pb
07:56:47.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:56:47.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
07:56:47.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
07:56:47.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
07:56:47.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
07:56:47.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:56:47.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:56:47.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:56:47.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
07:56:47.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
07:56:47.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.235.67:36093
07:56:47.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.25.235.67
07:56:47.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.25.235.126:46873,127.25.235.125:42159,127.25.235.124:39273
07:56:47.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=43721
07:56:47.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.235.67
07:56:47.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.235.84:44497
07:56:47.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:56:47.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:56:47.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
07:56:48.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:48.048053 27904 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.235.67:36093: connect: Connection refused (error 111) [suppressed 5 similar messages]
07:56:48.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:48.118270 28247 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
07:56:48.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:48.118582 28247 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
07:56:48.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:48.118671 28247 flags.cc:432] Enabled unsafe flag: --never_fsync=true
07:56:48.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:48.127818 28247 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
07:56:48.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:48.127995 28247 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.235.67
07:56:48.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.138476 28247 tablet_server_runner.cc:78] Tablet server non-default flags:
07:56:48.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:56:48.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.235.84:44497
07:56:48.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:56:48.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
07:56:48.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/data
07:56:48.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/wal
07:56:48.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:56:48.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:56:48.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.235.67:36093
07:56:48.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
07:56:48.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:56:48.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:56:48.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/data/info.pb
07:56:48.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.235.67
07:56:48.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=43721
07:56:48.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.25.235.126:46873,127.25.235.125:42159,127.25.235.124:39273
07:56:48.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
07:56:48.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:56:48.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
07:56:48.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
07:56:48.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:56:48.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.25.235.67
07:56:48.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/logs
07:56:48.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:56:48.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
07:56:48.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
07:56:48.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
07:56:48.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
07:56:48.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
07:56:48.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
07:56:48.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 07:43:22 UTC on bdcb31816ec0
07:56:48.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12770
07:56:48.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
07:56:48.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.140120 28247 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
07:56:48.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.142014 28247 file_cache.cc:504] Constructed file cache file cache with capacity 419430
07:56:48.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:48.152190 28252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:48.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:48.152482 28253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:48.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:48.152544 28255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:48.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.153493 28247 server_base.cc:1061] running on GCE node
07:56:48.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.154232 28247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
07:56:48.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.155434 28247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
07:56:48.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.156798 28247 hybrid_clock.cc:648] HybridClock initialized: now 1783497408156746 us; error 38 us; skew 500 ppm
07:56:48.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.157256 28247 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
07:56:48.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.159827 28247 webserver.cc:533] Webserver started at http://127.25.235.67:43721/ using document root <none> and password file <none>
07:56:48.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.160588 28247 fs_manager.cc:362] Metadata directory not provided
07:56:48.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.160698 28247 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
07:56:48.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.167148 28247 fs_manager.cc:714] Time spent opening directory manager: real 0.004s	user 0.005s	sys 0.000s
07:56:48.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.170323 28262 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
07:56:48.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.171751 28247 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
07:56:48.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.171983 28247 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/data,/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/wal
07:56:48.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "953ec1e7cb1345bd986b2b0171b6bcc2"
07:56:48.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 07:56:25 on dist-test-slave-63m1"
07:56:48.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.172814 28247 fs_report.cc:389] FS layout report
07:56:48.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
07:56:48.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/wal
07:56:48.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/wal
07:56:48.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/data/data
07:56:48.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
07:56:48.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
07:56:48.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
07:56:48.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
07:56:48.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
07:56:48.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
07:56:48.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
07:56:48.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
07:56:48.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
07:56:48.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
07:56:48.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
07:56:48.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.194322 28247 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
07:56:48.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.195537 28247 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
07:56:48.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.195936 28247 kserver.cc:163] Server-wide thread pool size limit: 3276
07:56:48.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.197154 28247 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
07:56:48.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.199844 28269 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
07:56:48.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.205873 28247 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
07:56:48.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.206027 28247 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.007s	user 0.001s	sys 0.000s
07:56:48.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.206200 28247 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
07:56:48.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.209703 28247 ts_tablet_manager.cc:616] Registered 1 tablets
07:56:48.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.209865 28247 ts_tablet_manager.cc:595] Time spent register tablets: real 0.004s	user 0.003s	sys 0.000s
07:56:48.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.210206 28269 tablet_bootstrap.cc:492] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2: Bootstrap starting.
07:56:48.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.236976 28269 log.cc:826] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2: Log is configured to *not* fsync() on all Append() calls
07:56:48.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.250124 28247 rpc_server.cc:307] RPC server started. Bound to: 127.25.235.67:36093
07:56:48.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.250172 28376 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.235.67:36093 every 8 connection(s)
07:56:48.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.251678 28247 server_base.cc:1193] Dumped server information to /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/data/info.pb
07:56:48.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.253314 26541 external_mini_cluster.cc:1434] Started /tmp/dist-test-task2rU0iV/build/asan/bin/kudu as pid 28247
07:56:48.254 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
07:56:48.279 [DEBUG - kudu-nio-7] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 21 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
07:56:48.280 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.25.235.65:44671
07:56:48.281 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

07:56:48.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.281529 26541 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-task2rU0iV/build/asan/bin/kudu pid 27885
07:56:48.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.317366 28011 generic_service.cc:196] Checking for leaks (request via RPC)
07:56:48.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.335928 28378 heartbeater.cc:344] Connected to a master server at 127.25.235.126:46873
07:56:48.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.336417 28378 heartbeater.cc:461] Registering TS with master...
07:56:48.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.337857 28378 heartbeater.cc:507] Master 127.25.235.126:46873 requested a full tablet report, sending...
07:56:48.341 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.341390 28377 heartbeater.cc:344] Connected to a master server at 127.25.235.124:39273
07:56:48.341 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.341444 26583 ts_manager.cc:194] Re-registered known tserver with Master: 953ec1e7cb1345bd986b2b0171b6bcc2 (127.25.235.67:36093)
07:56:48.341 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.341544 28377 heartbeater.cc:461] Registering TS with master...
07:56:48.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.342120 28377 heartbeater.cc:507] Master 127.25.235.124:39273 requested a full tablet report, sending...
07:56:48.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.343451 26724 ts_manager.cc:194] Re-registered known tserver with Master: 953ec1e7cb1345bd986b2b0171b6bcc2 (127.25.235.67:36093)
07:56:48.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.344823 26724 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.235.67:43545
07:56:48.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.345219 28269 tablet_bootstrap.cc:492] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=24 overwritten=0 applied=24 ignored=0} inserts{seen=18 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
07:56:48.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.345948 28269 tablet_bootstrap.cc:492] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2: Bootstrap complete.
07:56:48.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.347358 28269 ts_tablet_manager.cc:1403] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2: Time spent bootstrapping tablet: real 0.138s	user 0.042s	sys 0.021s
07:56:48.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.350660 28379 heartbeater.cc:344] Connected to a master server at 127.25.235.125:42159
07:56:48.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.350796 28379 heartbeater.cc:461] Registering TS with master...
07:56:48.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.351174 28379 heartbeater.cc:507] Master 127.25.235.125:42159 requested a full tablet report, sending...
07:56:48.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.352660 26654 ts_manager.cc:194] Re-registered known tserver with Master: 953ec1e7cb1345bd986b2b0171b6bcc2 (127.25.235.67:36093)
07:56:48.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.352906 28269 raft_consensus.cc:359] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [term 6 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:48.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.353813 28269 raft_consensus.cc:740] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 953ec1e7cb1345bd986b2b0171b6bcc2, State: Initialized, Role: FOLLOWER
07:56:48.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.354595 28269 consensus_queue.cc:260] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 24, Last appended: 6.24, Last appended by leader: 24, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:48.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.360123 28377 heartbeater.cc:499] Master 127.25.235.124:39273 was elected leader, sending a full tablet report...
07:56:48.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.360867 28269 ts_tablet_manager.cc:1434] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2: Time spent starting tablet: real 0.013s	user 0.003s	sys 0.011s
07:56:48.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.429617 26541 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task2rU0iV/build/asan/bin/kudu with pid 27885
07:56:48.448 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
07:56:48.448 [DEBUG - kudu-nio-7] (Connection.java:239) [peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)] handling channelInactive
07:56:48.448 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)] cleaning up while in state READY due to: connection closed
07:56:48.449 [DEBUG - Time-limited test] (Connection.java:748) [peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)] connecting to peer
07:56:48.449 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671
07:56:48.450 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671
07:56:48.450 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671) for tablet 9bc0a52dc27145398e9a7b4bdd4280b5: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671
07:56:48.462 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:48.462 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)[L],953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093),bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)]}]
07:56:48.462 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, attempt=2, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=14), Traces: [0ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [2ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [2ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [13ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=Deferred@1865616430(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f19483391e8@7de2b1c3 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6bf72153 -> wakeup thread Time-limited test)) after lookup
07:56:48.463 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)] connecting to peer
07:56:48.464 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671
07:56:48.464 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671
07:56:48.464 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671) for tablet 9bc0a52dc27145398e9a7b4bdd4280b5: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671
07:56:48.482 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:48.483 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)[L],953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093),bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)]}]
07:56:48.483 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, attempt=4, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=34), Traces: [0ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [2ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [2ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [13ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [15ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [16ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [16ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [34ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=Deferred@1865616430(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f19483391e8@7de2b1c3 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@579148199 after retry RPC@1881823517), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6bf72153 -> wakeup thread Time-limited test -> (continuation of Deferred@579148199 after retry RPC@1881823517))) after lookup
07:56:48.484 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)] connecting to peer
07:56:48.485 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671
07:56:48.485 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671
07:56:48.485 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671) for tablet 9bc0a52dc27145398e9a7b4bdd4280b5: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671
07:56:48.523 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:48.523 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)[L],953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093),bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)]}]
07:56:48.523 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, attempt=6, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=74), Traces: [0ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [2ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [2ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [13ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [15ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [16ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [16ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [34ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [35ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [37ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [37ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [71ms] refreshing cache from master, [71ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [74ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=Deferred@1865616430(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f19483391e8@7de2b1c3 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@579148199 after retry RPC@1881823517) -> (continuation of Deferred@1369328451 after retry RPC@244774712), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6bf72153 -> wakeup thread Time-limited test -> (continuation of Deferred@579148199 after retry RPC@1881823517) -> (continuation of Deferred@1369328451 after retry RPC@244774712))) after lookup
07:56:48.524 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)] connecting to peer
07:56:48.525 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671
07:56:48.525 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671
07:56:48.525 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671) for tablet 9bc0a52dc27145398e9a7b4bdd4280b5: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671
07:56:48.562 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:48.563 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)[L],953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093),bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)]}]
07:56:48.563 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, attempt=8, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=114), Traces: [0ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [2ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [2ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [13ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [15ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [16ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [16ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [34ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [35ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [37ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [37ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [71ms] refreshing cache from master, [71ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [74ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [76ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [77ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [77ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [111ms] refreshing cache from master, [111ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [114ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=Deferred@1865616430(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f19483391e8@7de2b1c3 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@579148199 after retry RPC@1881823517) -> (continuation of Deferred@1369328451 after retry RPC@244774712) -> (continuation of Deferred@229178351 after retry RPC@884925013), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6bf72153 -> wakeup thread Time-limited test -> (continuation of Deferred@579148199 after retry RPC@1881823517) -> (continuation of Deferred@1369328451 after retry RPC@244774712) -> (continuation of Deferred@229178351 after retry RPC@884925013))) after lookup
07:56:48.564 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)] connecting to peer
07:56:48.565 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671
07:56:48.565 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671
07:56:48.565 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671) for tablet 9bc0a52dc27145398e9a7b4bdd4280b5: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671
07:56:48.982 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:48.982 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)[L],953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093),bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)]}]
07:56:48.982 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, attempt=10, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=534), Traces: [0ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [2ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [2ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [13ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [15ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [16ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [16ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [34ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [35ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [37ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [37ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [71ms] refreshing cache from master, [71ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [74ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [76ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [77ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [77ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [111ms] refreshing cache from master, [111ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [114ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [115ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [117ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [117ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [530ms] refreshing cache from master, [531ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [533ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=Deferred@1865616430(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f19483391e8@7de2b1c3 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@579148199 after retry RPC@1881823517) -> (continuation of Deferred@1369328451 after retry RPC@244774712) -> (continuation of Deferred@229178351 after retry RPC@884925013) -> (continuation of Deferred@1579380580 after retry RPC@491557567), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6bf72153 -> wakeup thread Time-limited test -> (continuation of Deferred@579148199 after retry RPC@1881823517) -> (continuation of Deferred@1369328451 after retry RPC@244774712) -> (continuation of Deferred@229178351 after retry RPC@884925013) -> (continuation of Deferred@1579380580 after retry RPC@491557567))) after lookup
07:56:48.983 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)] connecting to peer
07:56:48.984 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671
07:56:48.985 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671
07:56:48.985 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671) for tablet 9bc0a52dc27145398e9a7b4bdd4280b5: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671
07:56:49.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:49.683416 28393 raft_consensus.cc:493] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3dd0f8e1dbda4e408e9caa288299ab5d)
07:56:49.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:49.683611 28393 raft_consensus.cc:515] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:49.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:49.685144 28393 leader_election.cc:290] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 953ec1e7cb1345bd986b2b0171b6bcc2 (127.25.235.67:36093), 3dd0f8e1dbda4e408e9caa288299ab5d (127.25.235.65:44671)
07:56:49.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:49.686358 28084 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.235.65:44671: connect: Connection refused (error 111)
07:56:49.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:49.687988 28084 leader_election.cc:336] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 3dd0f8e1dbda4e408e9caa288299ab5d (127.25.235.65:44671): Network error: Client connection negotiation failed: client connection to 127.25.235.65:44671: connect: Connection refused (error 111)
07:56:49.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:49.694577 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bc0a52dc27145398e9a7b4bdd4280b5" candidate_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" candidate_term: 8 candidate_status { last_received { term: 7 index: 28 } } ignore_live_leader: false dest_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" is_pre_election: true
07:56:49.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:49.695091 28331 raft_consensus.cc:2468] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd0c2de15db9438f9df3c22ffa0bed6c in term 6.
07:56:49.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:49.696208 28085 leader_election.cc:304] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 953ec1e7cb1345bd986b2b0171b6bcc2, bd0c2de15db9438f9df3c22ffa0bed6c; no voters: 3dd0f8e1dbda4e408e9caa288299ab5d
07:56:49.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:49.696720 28393 raft_consensus.cc:2804] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [term 7 FOLLOWER]: Leader pre-election won for term 8
07:56:49.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:49.696830 28393 raft_consensus.cc:493] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [term 7 FOLLOWER]: Starting leader election (detected failure of leader 3dd0f8e1dbda4e408e9caa288299ab5d)
07:56:49.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:49.697011 28393 raft_consensus.cc:3060] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [term 7 FOLLOWER]: Advancing to term 8
07:56:49.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:49.700145 28393 raft_consensus.cc:515] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:49.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:49.700992 28393 leader_election.cc:290] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [CANDIDATE]: Term 8 election: Requested vote from peers 953ec1e7cb1345bd986b2b0171b6bcc2 (127.25.235.67:36093), 3dd0f8e1dbda4e408e9caa288299ab5d (127.25.235.65:44671)
07:56:49.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:49.701673 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bc0a52dc27145398e9a7b4bdd4280b5" candidate_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" candidate_term: 8 candidate_status { last_received { term: 7 index: 28 } } ignore_live_leader: false dest_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2"
07:56:49.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:49.701910 28331 raft_consensus.cc:3060] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [term 6 FOLLOWER]: Advancing to term 8
07:56:49.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:49.702845 28084 leader_election.cc:336] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 3dd0f8e1dbda4e408e9caa288299ab5d (127.25.235.65:44671): Network error: Client connection negotiation failed: client connection to 127.25.235.65:44671: connect: Connection refused (error 111)
07:56:49.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:49.706079 28331 raft_consensus.cc:2468] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd0c2de15db9438f9df3c22ffa0bed6c in term 8.
07:56:49.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:49.706852 28085 leader_election.cc:304] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 953ec1e7cb1345bd986b2b0171b6bcc2, bd0c2de15db9438f9df3c22ffa0bed6c; no voters: 3dd0f8e1dbda4e408e9caa288299ab5d
07:56:49.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:49.707234 28393 raft_consensus.cc:2804] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [term 8 FOLLOWER]: Leader election won for term 8
07:56:49.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:49.707707 28393 raft_consensus.cc:697] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [term 8 LEADER]: Becoming Leader. State: Replica: bd0c2de15db9438f9df3c22ffa0bed6c, State: Running, Role: LEADER
07:56:49.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:49.708349 28393 consensus_queue.cc:237] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 7.28, Last appended by leader: 28, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:49.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:49.712805 26724 catalog_manager.cc:5697] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c reported cstate change: term changed from 7 to 8, leader changed from 3dd0f8e1dbda4e408e9caa288299ab5d (127.25.235.65) to bd0c2de15db9438f9df3c22ffa0bed6c (127.25.235.66). New cstate: current_term: 8 leader_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } health_report { overall_health: HEALTHY } } }
07:56:50.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.128561 28331 raft_consensus.cc:1275] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [term 8 FOLLOWER]: Refusing update from remote peer bd0c2de15db9438f9df3c22ffa0bed6c: Log matching property violated. Preceding OpId in replica: term: 6 index: 24. Preceding OpId from leader: term: 8 index: 29. (index mismatch)
07:56:50.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.129648 28393 consensus_queue.cc:1048] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 24, Time since last communication: 0.000s
07:56:50.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:50.150555 28084 consensus_peers.cc:597] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c -> Peer 3dd0f8e1dbda4e408e9caa288299ab5d (127.25.235.65:44671): Couldn't send request to peer 3dd0f8e1dbda4e408e9caa288299ab5d. Status: Network error: Client connection negotiation failed: client connection to 127.25.235.65:44671: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
07:56:50.162 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:50.162 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671),953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093),bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)[L]]}]
07:56:50.163 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, attempt=12, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=1714), Traces: [0ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [2ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [2ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [13ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [15ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [16ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [16ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [34ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [35ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [37ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [37ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [71ms] refreshing cache from master, [71ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [74ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [76ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [77ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [77ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [111ms] refreshing cache from master, [111ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [114ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [115ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [117ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [117ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [530ms] refreshing cache from master, [531ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [533ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [535ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [537ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [537ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [1710ms] refreshing cache from master, [1711ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [1713ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=Deferred@1865616430(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f19483391e8@7de2b1c3 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@579148199 after retry RPC@1881823517) -> (continuation of Deferred@1369328451 after retry RPC@244774712) -> (continuation of Deferred@229178351 after retry RPC@884925013) -> (continuation of Deferred@1579380580 after retry RPC@491557567) -> (continuation of Deferred@393712794 after retry RPC@1859597729), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6bf72153 -> wakeup thread Time-limited test -> (continuation of Deferred@579148199 after retry RPC@1881823517) -> (continuation of Deferred@1369328451 after retry RPC@244774712) -> (continuation of Deferred@229178351 after retry RPC@884925013) -> (continuation of Deferred@1579380580 after retry RPC@491557567) -> (continuation of Deferred@393712794 after retry RPC@1859597729))) after lookup
07:56:50.163 [DEBUG - kudu-nio-2] (Connection.java:748) [peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)] connecting to peer
07:56:50.164 [DEBUG - kudu-nio-6] (Connection.java:757) [peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)] Successfully connected to peer
07:56:50.190 [DEBUG - kudu-nio-6] (Negotiator.java:918) Authenticated connection [id: 0xa2db6e59, L:/127.0.0.1:44510 - R:/127.25.235.66:42989] using TOKEN/null
07:56:50.218 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.25.235.65:44671
07:56:50.218 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

07:56:50.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.219218 26541 external_mini_cluster.cc:1372] Running /tmp/dist-test-task2rU0iV/build/asan/bin/kudu
07:56:50.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task2rU0iV/build/asan/bin/kudu
07:56:50.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-0/wal
07:56:50.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-0/data
07:56:50.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
07:56:50.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
07:56:50.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
07:56:50.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:56:50.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:56:50.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:56:50.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-0/logs
07:56:50.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-0/data/info.pb
07:56:50.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:56:50.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
07:56:50.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
07:56:50.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
07:56:50.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
07:56:50.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:56:50.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:56:50.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:56:50.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
07:56:50.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
07:56:50.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.235.65:44671
07:56:50.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.25.235.65
07:56:50.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.25.235.126:46873,127.25.235.125:42159,127.25.235.124:39273
07:56:50.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42151
07:56:50.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.235.65
07:56:50.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.235.84:44497
07:56:50.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:56:50.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:56:50.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
07:56:50.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:50.579823 28420 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
07:56:50.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:50.580255 28420 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
07:56:50.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:50.580374 28420 flags.cc:432] Enabled unsafe flag: --never_fsync=true
07:56:50.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:50.589895 28420 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
07:56:50.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:50.590119 28420 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.235.65
07:56:50.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.601110 28420 tablet_server_runner.cc:78] Tablet server non-default flags:
07:56:50.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:56:50.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.235.84:44497
07:56:50.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:56:50.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
07:56:50.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-0/data
07:56:50.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-0/wal
07:56:50.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:56:50.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:56:50.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.235.65:44671
07:56:50.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
07:56:50.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:56:50.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:56:50.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-0/data/info.pb
07:56:50.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.235.65
07:56:50.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42151
07:56:50.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.25.235.126:46873,127.25.235.125:42159,127.25.235.124:39273
07:56:50.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
07:56:50.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:56:50.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
07:56:50.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
07:56:50.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:56:50.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.25.235.65
07:56:50.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-0/logs
07:56:50.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:56:50.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
07:56:50.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
07:56:50.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
07:56:50.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
07:56:50.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
07:56:50.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
07:56:50.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 07:43:22 UTC on bdcb31816ec0
07:56:50.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12770
07:56:50.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
07:56:50.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.603534 28420 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
07:56:50.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.605515 28420 file_cache.cc:504] Constructed file cache file cache with capacity 419430
07:56:50.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:50.615994 28425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:50.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:50.615952 28426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:50.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:50.616869 28428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:50.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.617753 28420 server_base.cc:1061] running on GCE node
07:56:50.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.618340 28420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
07:56:50.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.619266 28420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
07:56:50.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.620456 28420 hybrid_clock.cc:648] HybridClock initialized: now 1783497410620372 us; error 83 us; skew 500 ppm
07:56:50.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.620738 28420 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
07:56:50.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.622900 28420 webserver.cc:533] Webserver started at http://127.25.235.65:42151/ using document root <none> and password file <none>
07:56:50.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.623613 28420 fs_manager.cc:362] Metadata directory not provided
07:56:50.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.623701 28420 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
07:56:50.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.629416 28420 fs_manager.cc:714] Time spent opening directory manager: real 0.004s	user 0.002s	sys 0.001s
07:56:50.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.632488 28434 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
07:56:50.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.634100 28420 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
07:56:50.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.634241 28420 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-0/data,/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-0/wal
07:56:50.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "3dd0f8e1dbda4e408e9caa288299ab5d"
07:56:50.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 07:56:24 on dist-test-slave-63m1"
07:56:50.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.634917 28420 fs_report.cc:389] FS layout report
07:56:50.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
07:56:50.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-0/wal
07:56:50.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-0/wal
07:56:50.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-0/data/data
07:56:50.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
07:56:50.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
07:56:50.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
07:56:50.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
07:56:50.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
07:56:50.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
07:56:50.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
07:56:50.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
07:56:50.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
07:56:50.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
07:56:50.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
07:56:50.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.658080 28420 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
07:56:50.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.658914 28420 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
07:56:50.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.659158 28420 kserver.cc:163] Server-wide thread pool size limit: 3276
07:56:50.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.660081 28420 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
07:56:50.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.662240 28441 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
07:56:50.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.668315 28420 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
07:56:50.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.668412 28420 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.007s	user 0.001s	sys 0.000s
07:56:50.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.668517 28420 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
07:56:50.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.671559 28420 ts_tablet_manager.cc:616] Registered 1 tablets
07:56:50.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.671728 28420 ts_tablet_manager.cc:595] Time spent register tablets: real 0.003s	user 0.001s	sys 0.000s
07:56:50.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.672235 28441 tablet_bootstrap.cc:492] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d: Bootstrap starting.
07:56:50.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.699462 28441 log.cc:826] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d: Log is configured to *not* fsync() on all Append() calls
07:56:50.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.727507 28420 rpc_server.cc:307] RPC server started. Bound to: 127.25.235.65:44671
07:56:50.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.727572 28548 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.235.65:44671 every 8 connection(s)
07:56:50.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.729323 28420 server_base.cc:1193] Dumped server information to /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-0/data/info.pb
07:56:50.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.738199 26541 external_mini_cluster.cc:1434] Started /tmp/dist-test-task2rU0iV/build/asan/bin/kudu as pid 28420
07:56:50.739 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
07:56:50.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.740628 28441 tablet_bootstrap.cc:492] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=21 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
07:56:50.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.741500 28441 tablet_bootstrap.cc:492] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d: Bootstrap complete.
07:56:50.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.743278 28441 ts_tablet_manager.cc:1403] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d: Time spent bootstrapping tablet: real 0.071s	user 0.050s	sys 0.012s
07:56:50.761 [DEBUG - kudu-nio-6] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 24 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
07:56:50.762 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.25.235.66:42989
07:56:50.763 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

07:56:50.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.763454 26541 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-task2rU0iV/build/asan/bin/kudu pid 28068
07:56:50.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.774883 28441 raft_consensus.cc:359] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 7 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:50.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.775975 28441 raft_consensus.cc:740] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dd0f8e1dbda4e408e9caa288299ab5d, State: Initialized, Role: FOLLOWER
07:56:50.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.776850 28441 consensus_queue.cc:260] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 7.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:50.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.788582 28441 ts_tablet_manager.cc:1434] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d: Time spent starting tablet: real 0.045s	user 0.006s	sys 0.007s
07:56:50.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.789610 28549 heartbeater.cc:344] Connected to a master server at 127.25.235.124:39273
07:56:50.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.790467 28549 heartbeater.cc:461] Registering TS with master...
07:56:50.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.789695 28550 heartbeater.cc:344] Connected to a master server at 127.25.235.126:46873
07:56:50.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.791014 28550 heartbeater.cc:461] Registering TS with master...
07:56:50.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.791776 28550 heartbeater.cc:507] Master 127.25.235.126:46873 requested a full tablet report, sending...
07:56:50.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.791787 28549 heartbeater.cc:507] Master 127.25.235.124:39273 requested a full tablet report, sending...
07:56:50.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.795224 26725 ts_manager.cc:194] Re-registered known tserver with Master: 3dd0f8e1dbda4e408e9caa288299ab5d (127.25.235.65:44671)
07:56:50.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.796706 26725 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.235.65:40653
07:56:50.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.799389 26583 ts_manager.cc:194] Re-registered known tserver with Master: 3dd0f8e1dbda4e408e9caa288299ab5d (127.25.235.65:44671)
07:56:50.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.800738 28549 heartbeater.cc:499] Master 127.25.235.124:39273 was elected leader, sending a full tablet report...
07:56:50.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.804188 28551 heartbeater.cc:344] Connected to a master server at 127.25.235.125:42159
07:56:50.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.804339 28551 heartbeater.cc:461] Registering TS with master...
07:56:50.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.804652 28551 heartbeater.cc:507] Master 127.25.235.125:42159 requested a full tablet report, sending...
07:56:50.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.805107 28192 generic_service.cc:196] Checking for leaks (request via RPC)
07:56:50.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.805946 26654 ts_manager.cc:194] Re-registered known tserver with Master: 3dd0f8e1dbda4e408e9caa288299ab5d (127.25.235.65:44671)
07:56:50.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.907775 26541 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task2rU0iV/build/asan/bin/kudu with pid 28068
07:56:50.922 [DEBUG - kudu-nio-6] (Connection.java:239) [peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)] handling channelInactive
07:56:50.922 [DEBUG - kudu-nio-6] (Connection.java:709) [peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)] cleaning up while in state READY due to: connection closed
07:56:50.923 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
07:56:50.923 [DEBUG - Time-limited test] (Connection.java:748) [peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)] connecting to peer
07:56:50.924 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989
07:56:50.925 [DEBUG - kudu-nio-7] (Connection.java:709) [peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)] cleaning up while in state CONNECTING due to: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989
07:56:50.925 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989) for tablet 9bc0a52dc27145398e9a7b4bdd4280b5: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989
07:56:50.942 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:50.942 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671),953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093),bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)[L]]}]
07:56:50.942 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, attempt=2, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=19), Traces: [0ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [2ms] received response from server bd0c2de15db9438f9df3c22ffa0bed6c: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [2ms] delaying RPC due to: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [16ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [19ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=Deferred@834742525(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f19483391e8@66ec6959 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2d220315 -> wakeup thread Time-limited test)) after lookup
07:56:50.943 [DEBUG - kudu-nio-2] (Connection.java:748) [peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)] connecting to peer
07:56:50.945 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989
07:56:50.945 [DEBUG - kudu-nio-0] (Connection.java:709) [peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)] cleaning up while in state CONNECTING due to: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989
07:56:50.945 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989) for tablet 9bc0a52dc27145398e9a7b4bdd4280b5: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989
07:56:50.963 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:50.963 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671),953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093),bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)[L]]}]
07:56:50.963 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, attempt=4, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=40), Traces: [0ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [2ms] received response from server bd0c2de15db9438f9df3c22ffa0bed6c: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [2ms] delaying RPC due to: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [16ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [19ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [20ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [22ms] received response from server bd0c2de15db9438f9df3c22ffa0bed6c: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [22ms] delaying RPC due to: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [39ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=Deferred@834742525(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f19483391e8@66ec6959 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1194464923 after retry RPC@2058719171), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2d220315 -> wakeup thread Time-limited test -> (continuation of Deferred@1194464923 after retry RPC@2058719171))) after lookup
07:56:50.964 [DEBUG - kudu-nio-2] (Connection.java:748) [peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)] connecting to peer
07:56:50.965 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989
07:56:50.965 [DEBUG - kudu-nio-1] (Connection.java:709) [peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)] cleaning up while in state CONNECTING due to: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989
07:56:50.965 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989) for tablet 9bc0a52dc27145398e9a7b4bdd4280b5: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989
07:56:50.982 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:50.982 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671),953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093),bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)[L]]}]
07:56:50.982 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, attempt=6, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=59), Traces: [0ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [2ms] received response from server bd0c2de15db9438f9df3c22ffa0bed6c: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [2ms] delaying RPC due to: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [16ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [19ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [20ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [22ms] received response from server bd0c2de15db9438f9df3c22ffa0bed6c: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [22ms] delaying RPC due to: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [39ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [40ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [42ms] received response from server bd0c2de15db9438f9df3c22ffa0bed6c: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [42ms] delaying RPC due to: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [56ms] refreshing cache from master, [56ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [59ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=Deferred@834742525(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f19483391e8@66ec6959 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1194464923 after retry RPC@2058719171) -> (continuation of Deferred@1742454115 after retry RPC@826736076), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2d220315 -> wakeup thread Time-limited test -> (continuation of Deferred@1194464923 after retry RPC@2058719171) -> (continuation of Deferred@1742454115 after retry RPC@826736076))) after lookup
07:56:50.983 [DEBUG - kudu-nio-2] (Connection.java:748) [peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)] connecting to peer
07:56:50.984 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989
07:56:50.984 [DEBUG - kudu-nio-2] (Connection.java:709) [peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)] cleaning up while in state CONNECTING due to: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989
07:56:50.984 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989) for tablet 9bc0a52dc27145398e9a7b4bdd4280b5: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989
07:56:51.101 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:51.102 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671),953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093),bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)[L]]}]
07:56:51.102 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, attempt=8, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=178), Traces: [0ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [2ms] received response from server bd0c2de15db9438f9df3c22ffa0bed6c: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [2ms] delaying RPC due to: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [16ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [19ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [20ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [22ms] received response from server bd0c2de15db9438f9df3c22ffa0bed6c: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [22ms] delaying RPC due to: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [39ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [40ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [42ms] received response from server bd0c2de15db9438f9df3c22ffa0bed6c: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [42ms] delaying RPC due to: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [56ms] refreshing cache from master, [56ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [59ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [60ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [61ms] received response from server bd0c2de15db9438f9df3c22ffa0bed6c: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [61ms] delaying RPC due to: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [175ms] refreshing cache from master, [176ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [178ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=Deferred@834742525(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f19483391e8@66ec6959 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1194464923 after retry RPC@2058719171) -> (continuation of Deferred@1742454115 after retry RPC@826736076) -> (continuation of Deferred@88458912 after retry RPC@283341977), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2d220315 -> wakeup thread Time-limited test -> (continuation of Deferred@1194464923 after retry RPC@2058719171) -> (continuation of Deferred@1742454115 after retry RPC@826736076) -> (continuation of Deferred@88458912 after retry RPC@283341977))) after lookup
07:56:51.102 [DEBUG - kudu-nio-2] (Connection.java:748) [peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)] connecting to peer
07:56:51.103 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989
07:56:51.103 [DEBUG - kudu-nio-3] (Connection.java:709) [peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)] cleaning up while in state CONNECTING due to: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989
07:56:51.104 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989) for tablet 9bc0a52dc27145398e9a7b4bdd4280b5: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989
07:56:51.562 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:51.563 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671),953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093),bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)[L]]}]
07:56:51.563 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, attempt=10, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=640), Traces: [0ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [2ms] received response from server bd0c2de15db9438f9df3c22ffa0bed6c: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [2ms] delaying RPC due to: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [16ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [19ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [20ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [22ms] received response from server bd0c2de15db9438f9df3c22ffa0bed6c: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [22ms] delaying RPC due to: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [39ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [40ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [42ms] received response from server bd0c2de15db9438f9df3c22ffa0bed6c: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [42ms] delaying RPC due to: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [56ms] refreshing cache from master, [56ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [59ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [60ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [61ms] received response from server bd0c2de15db9438f9df3c22ffa0bed6c: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [61ms] delaying RPC due to: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [175ms] refreshing cache from master, [176ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [178ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [179ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [181ms] received response from server bd0c2de15db9438f9df3c22ffa0bed6c: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [181ms] delaying RPC due to: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [636ms] refreshing cache from master, [636ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [639ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=Deferred@834742525(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f19483391e8@66ec6959 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1194464923 after retry RPC@2058719171) -> (continuation of Deferred@1742454115 after retry RPC@826736076) -> (continuation of Deferred@88458912 after retry RPC@283341977) -> (continuation of Deferred@1285293383 after retry RPC@1196484593), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2d220315 -> wakeup thread Time-limited test -> (continuation of Deferred@1194464923 after retry RPC@2058719171) -> (continuation of Deferred@1742454115 after retry RPC@826736076) -> (continuation of Deferred@88458912 after retry RPC@283341977) -> (continuation of Deferred@1285293383 after retry RPC@1196484593))) after lookup
07:56:51.564 [DEBUG - kudu-nio-2] (Connection.java:748) [peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)] connecting to peer
07:56:51.566 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989
07:56:51.566 [DEBUG - kudu-nio-4] (Connection.java:709) [peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)] cleaning up while in state CONNECTING due to: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989
07:56:51.566 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989) for tablet 9bc0a52dc27145398e9a7b4bdd4280b5: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989
07:56:52.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:52.019337 28564 raft_consensus.cc:493] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
07:56:52.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:52.019639 28564 raft_consensus.cc:515] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:52.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:52.021521 28564 leader_election.cc:290] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 953ec1e7cb1345bd986b2b0171b6bcc2 (127.25.235.67:36093), bd0c2de15db9438f9df3c22ffa0bed6c (127.25.235.66:42989)
07:56:52.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:52.035293 28436 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.235.66:42989: connect: Connection refused (error 111)
07:56:52.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:52.037173 28436 leader_election.cc:336] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer bd0c2de15db9438f9df3c22ffa0bed6c (127.25.235.66:42989): Network error: Client connection negotiation failed: client connection to 127.25.235.66:42989: connect: Connection refused (error 111)
07:56:52.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:52.042291 28331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bc0a52dc27145398e9a7b4bdd4280b5" candidate_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" candidate_term: 8 candidate_status { last_received { term: 7 index: 28 } } ignore_live_leader: false dest_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" is_pre_election: true
07:56:52.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:52.043398 28437 leader_election.cc:304] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3dd0f8e1dbda4e408e9caa288299ab5d; no voters: 953ec1e7cb1345bd986b2b0171b6bcc2, bd0c2de15db9438f9df3c22ffa0bed6c
07:56:52.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:52.043989 28564 raft_consensus.cc:2749] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
07:56:52.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:52.195220 28570 raft_consensus.cc:493] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader bd0c2de15db9438f9df3c22ffa0bed6c)
07:56:52.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:52.195503 28570 raft_consensus.cc:515] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:52.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:52.197359 28570 leader_election.cc:290] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3dd0f8e1dbda4e408e9caa288299ab5d (127.25.235.65:44671), bd0c2de15db9438f9df3c22ffa0bed6c (127.25.235.66:42989)
07:56:52.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:52.199186 28264 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.235.66:42989: connect: Connection refused (error 111)
07:56:52.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:52.200826 28264 leader_election.cc:336] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer bd0c2de15db9438f9df3c22ffa0bed6c (127.25.235.66:42989): Network error: Client connection negotiation failed: client connection to 127.25.235.66:42989: connect: Connection refused (error 111)
07:56:52.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:52.208091 28503 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bc0a52dc27145398e9a7b4bdd4280b5" candidate_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 32 } } ignore_live_leader: false dest_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" is_pre_election: true
07:56:52.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:52.208528 28503 raft_consensus.cc:2468] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 953ec1e7cb1345bd986b2b0171b6bcc2 in term 7.
07:56:52.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:52.209568 28264 leader_election.cc:304] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3dd0f8e1dbda4e408e9caa288299ab5d, 953ec1e7cb1345bd986b2b0171b6bcc2; no voters: bd0c2de15db9438f9df3c22ffa0bed6c
07:56:52.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:52.210016 28570 raft_consensus.cc:2804] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [term 8 FOLLOWER]: Leader pre-election won for term 9
07:56:52.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:52.210158 28570 raft_consensus.cc:493] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [term 8 FOLLOWER]: Starting leader election (detected failure of leader bd0c2de15db9438f9df3c22ffa0bed6c)
07:56:52.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:52.210244 28570 raft_consensus.cc:3060] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [term 8 FOLLOWER]: Advancing to term 9
07:56:52.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:52.213094 28570 raft_consensus.cc:515] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:52.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:52.213831 28570 leader_election.cc:290] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [CANDIDATE]: Term 9 election: Requested vote from peers 3dd0f8e1dbda4e408e9caa288299ab5d (127.25.235.65:44671), bd0c2de15db9438f9df3c22ffa0bed6c (127.25.235.66:42989)
07:56:52.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:52.214493 28503 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bc0a52dc27145398e9a7b4bdd4280b5" candidate_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 32 } } ignore_live_leader: false dest_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d"
07:56:52.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:52.214766 28503 raft_consensus.cc:3060] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 7 FOLLOWER]: Advancing to term 9
07:56:52.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:52.215685 28264 leader_election.cc:336] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer bd0c2de15db9438f9df3c22ffa0bed6c (127.25.235.66:42989): Network error: Client connection negotiation failed: client connection to 127.25.235.66:42989: connect: Connection refused (error 111)
07:56:52.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:52.219089 28503 raft_consensus.cc:2468] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 953ec1e7cb1345bd986b2b0171b6bcc2 in term 9.
07:56:52.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:52.219612 28264 leader_election.cc:304] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3dd0f8e1dbda4e408e9caa288299ab5d, 953ec1e7cb1345bd986b2b0171b6bcc2; no voters: bd0c2de15db9438f9df3c22ffa0bed6c
07:56:52.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:52.220036 28570 raft_consensus.cc:2804] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [term 9 FOLLOWER]: Leader election won for term 9
07:56:52.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:52.220496 28570 raft_consensus.cc:697] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [term 9 LEADER]: Becoming Leader. State: Replica: 953ec1e7cb1345bd986b2b0171b6bcc2, State: Running, Role: LEADER
07:56:52.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:52.221071 28570 consensus_queue.cc:237] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 8.32, Last appended by leader: 32, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:52.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:52.225273 26725 catalog_manager.cc:5697] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 reported cstate change: term changed from 8 to 9, leader changed from bd0c2de15db9438f9df3c22ffa0bed6c (127.25.235.66) to 953ec1e7cb1345bd986b2b0171b6bcc2 (127.25.235.67). New cstate: current_term: 9 leader_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } health_report { overall_health: UNKNOWN } } }
07:56:52.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:52.607157 28503 raft_consensus.cc:1275] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 9 FOLLOWER]: Refusing update from remote peer 953ec1e7cb1345bd986b2b0171b6bcc2: Log matching property violated. Preceding OpId in replica: term: 7 index: 28. Preceding OpId from leader: term: 9 index: 33. (index mismatch)
07:56:52.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:52.608276 28570 consensus_queue.cc:1048] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 28, Time since last communication: 0.000s
07:56:52.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:52.624507 28264 consensus_peers.cc:597] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 -> Peer bd0c2de15db9438f9df3c22ffa0bed6c (127.25.235.66:42989): Couldn't send request to peer bd0c2de15db9438f9df3c22ffa0bed6c. Status: Network error: Client connection negotiation failed: client connection to 127.25.235.66:42989: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
07:56:53.182 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:53.182 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671),953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)[L],bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)]}]
07:56:53.183 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, attempt=12, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=2259), Traces: [0ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [2ms] received response from server bd0c2de15db9438f9df3c22ffa0bed6c: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [2ms] delaying RPC due to: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [16ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [19ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [20ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [22ms] received response from server bd0c2de15db9438f9df3c22ffa0bed6c: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [22ms] delaying RPC due to: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [39ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [40ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [42ms] received response from server bd0c2de15db9438f9df3c22ffa0bed6c: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [42ms] delaying RPC due to: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [56ms] refreshing cache from master, [56ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [59ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [60ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [61ms] received response from server bd0c2de15db9438f9df3c22ffa0bed6c: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [61ms] delaying RPC due to: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [175ms] refreshing cache from master, [176ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [178ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [179ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [181ms] received response from server bd0c2de15db9438f9df3c22ffa0bed6c: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [181ms] delaying RPC due to: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [636ms] refreshing cache from master, [636ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [639ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [641ms] sending RPC to server bd0c2de15db9438f9df3c22ffa0bed6c, [643ms] received response from server bd0c2de15db9438f9df3c22ffa0bed6c: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [643ms] delaying RPC due to: Network error: Failed to connect to peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989): Connection refused: /127.25.235.66:42989, [2256ms] refreshing cache from master, [2256ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [2259ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=Deferred@834742525(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f19483391e8@66ec6959 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1194464923 after retry RPC@2058719171) -> (continuation of Deferred@1742454115 after retry RPC@826736076) -> (continuation of Deferred@88458912 after retry RPC@283341977) -> (continuation of Deferred@1285293383 after retry RPC@1196484593) -> (continuation of Deferred@1256163667 after retry RPC@175149206), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2d220315 -> wakeup thread Time-limited test -> (continuation of Deferred@1194464923 after retry RPC@2058719171) -> (continuation of Deferred@1742454115 after retry RPC@826736076) -> (continuation of Deferred@88458912 after retry RPC@283341977) -> (continuation of Deferred@1285293383 after retry RPC@1196484593) -> (continuation of Deferred@1256163667 after retry RPC@175149206))) after lookup
07:56:53.183 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)] connecting to peer
07:56:53.185 [DEBUG - kudu-nio-5] (Connection.java:757) [peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)] Successfully connected to peer
07:56:53.207 [DEBUG - kudu-nio-5] (Negotiator.java:918) Authenticated connection [id: 0x92053954, L:/127.0.0.1:33764 - R:/127.25.235.67:36093] using TOKEN/null
07:56:53.232 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.25.235.66:42989
07:56:53.233 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

07:56:53.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.233722 26541 external_mini_cluster.cc:1372] Running /tmp/dist-test-task2rU0iV/build/asan/bin/kudu
07:56:53.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task2rU0iV/build/asan/bin/kudu
07:56:53.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-1/wal
07:56:53.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-1/data
07:56:53.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
07:56:53.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
07:56:53.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
07:56:53.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:56:53.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:56:53.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:56:53.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-1/logs
07:56:53.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-1/data/info.pb
07:56:53.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:56:53.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
07:56:53.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
07:56:53.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
07:56:53.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
07:56:53.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:56:53.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:56:53.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:56:53.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
07:56:53.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
07:56:53.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.235.66:42989
07:56:53.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.25.235.66
07:56:53.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.25.235.126:46873,127.25.235.125:42159,127.25.235.124:39273
07:56:53.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42391
07:56:53.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.235.66
07:56:53.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.235.84:44497
07:56:53.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:56:53.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:56:53.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
07:56:53.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:53.585180 28601 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
07:56:53.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:53.585587 28601 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
07:56:53.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:53.585731 28601 flags.cc:432] Enabled unsafe flag: --never_fsync=true
07:56:53.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:53.596196 28601 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
07:56:53.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:53.596473 28601 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.235.66
07:56:53.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.608109 28601 tablet_server_runner.cc:78] Tablet server non-default flags:
07:56:53.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:56:53.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.235.84:44497
07:56:53.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:56:53.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
07:56:53.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-1/data
07:56:53.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-1/wal
07:56:53.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:56:53.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:56:53.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.235.66:42989
07:56:53.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
07:56:53.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:56:53.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:56:53.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-1/data/info.pb
07:56:53.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.235.66
07:56:53.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42391
07:56:53.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.25.235.126:46873,127.25.235.125:42159,127.25.235.124:39273
07:56:53.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
07:56:53.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:56:53.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
07:56:53.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
07:56:53.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:56:53.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.25.235.66
07:56:53.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-1/logs
07:56:53.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:56:53.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
07:56:53.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
07:56:53.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
07:56:53.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
07:56:53.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
07:56:53.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
07:56:53.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 07:43:22 UTC on bdcb31816ec0
07:56:53.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12770
07:56:53.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
07:56:53.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.610390 28601 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
07:56:53.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.612387 28601 file_cache.cc:504] Constructed file cache file cache with capacity 419430
07:56:53.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:53.622066 28607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:53.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:53.622066 28606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:53.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:53.623009 28609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:53.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.624290 28601 server_base.cc:1061] running on GCE node
07:56:53.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.625126 28601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
07:56:53.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.627035 28601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
07:56:53.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.628391 28601 hybrid_clock.cc:648] HybridClock initialized: now 1783497413628315 us; error 57 us; skew 500 ppm
07:56:53.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.628818 28601 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
07:56:53.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.631583 28601 webserver.cc:533] Webserver started at http://127.25.235.66:42391/ using document root <none> and password file <none>
07:56:53.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.632513 28601 fs_manager.cc:362] Metadata directory not provided
07:56:53.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.632606 28601 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
07:56:53.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.638214 28601 fs_manager.cc:714] Time spent opening directory manager: real 0.004s	user 0.001s	sys 0.003s
07:56:53.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.641389 28615 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
07:56:53.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.642781 28601 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.002s
07:56:53.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.642946 28601 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-1/data,/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-1/wal
07:56:53.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "bd0c2de15db9438f9df3c22ffa0bed6c"
07:56:53.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 07:56:24 on dist-test-slave-63m1"
07:56:53.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.643641 28601 fs_report.cc:389] FS layout report
07:56:53.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
07:56:53.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-1/wal
07:56:53.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-1/wal
07:56:53.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-1/data/data
07:56:53.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
07:56:53.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
07:56:53.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
07:56:53.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
07:56:53.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
07:56:53.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
07:56:53.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
07:56:53.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
07:56:53.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
07:56:53.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
07:56:53.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
07:56:53.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.660609 28601 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
07:56:53.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.661584 28601 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
07:56:53.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.661885 28601 kserver.cc:163] Server-wide thread pool size limit: 3276
07:56:53.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.662839 28601 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
07:56:53.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.665094 28622 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
07:56:53.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.671118 28601 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
07:56:53.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.671232 28601 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.007s	user 0.001s	sys 0.000s
07:56:53.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.671344 28601 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
07:56:53.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.674680 28601 ts_tablet_manager.cc:616] Registered 1 tablets
07:56:53.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.674787 28601 ts_tablet_manager.cc:595] Time spent register tablets: real 0.003s	user 0.002s	sys 0.000s
07:56:53.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.675305 28622 tablet_bootstrap.cc:492] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c: Bootstrap starting.
07:56:53.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.702724 28622 log.cc:826] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c: Log is configured to *not* fsync() on all Append() calls
07:56:53.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.723258 28601 rpc_server.cc:307] RPC server started. Bound to: 127.25.235.66:42989
07:56:53.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.723270 28731 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.235.66:42989 every 8 connection(s)
07:56:53.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.725297 28601 server_base.cc:1193] Dumped server information to /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-1/data/info.pb
07:56:53.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.732530 26541 external_mini_cluster.cc:1434] Started /tmp/dist-test-task2rU0iV/build/asan/bin/kudu as pid 28601
07:56:53.733 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
07:56:53.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.758502 28622 tablet_bootstrap.cc:492] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=24 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
07:56:53.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.759662 28622 tablet_bootstrap.cc:492] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c: Bootstrap complete.
07:56:53.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.761960 28622 ts_tablet_manager.cc:1403] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c: Time spent bootstrapping tablet: real 0.087s	user 0.068s	sys 0.016s
07:56:53.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.774102 28622 raft_consensus.cc:359] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [term 8 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:53.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.775669 28622 raft_consensus.cc:740] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd0c2de15db9438f9df3c22ffa0bed6c, State: Initialized, Role: FOLLOWER
07:56:53.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.776705 28622 consensus_queue.cc:260] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 8.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:53.783 [DEBUG - kudu-nio-5] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 27 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
07:56:53.785 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.25.235.67:36093
07:56:53.785 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

07:56:53.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.785557 28622 ts_tablet_manager.cc:1434] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c: Time spent starting tablet: real 0.023s	user 0.017s	sys 0.006s
07:56:53.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.785763 26541 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-task2rU0iV/build/asan/bin/kudu pid 28247
07:56:53.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.805212 28732 heartbeater.cc:344] Connected to a master server at 127.25.235.124:39273
07:56:53.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.805265 28733 heartbeater.cc:344] Connected to a master server at 127.25.235.126:46873
07:56:53.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.805595 28732 heartbeater.cc:461] Registering TS with master...
07:56:53.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.805646 28734 heartbeater.cc:344] Connected to a master server at 127.25.235.125:42159
07:56:53.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.805816 28734 heartbeater.cc:461] Registering TS with master...
07:56:53.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.805593 28733 heartbeater.cc:461] Registering TS with master...
07:56:53.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.806893 28732 heartbeater.cc:507] Master 127.25.235.124:39273 requested a full tablet report, sending...
07:56:53.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.806957 28734 heartbeater.cc:507] Master 127.25.235.125:42159 requested a full tablet report, sending...
07:56:53.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.807113 28733 heartbeater.cc:507] Master 127.25.235.126:46873 requested a full tablet report, sending...
07:56:53.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.811834 26583 ts_manager.cc:194] Re-registered known tserver with Master: bd0c2de15db9438f9df3c22ffa0bed6c (127.25.235.66:42989)
07:56:53.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.815400 28372 generic_service.cc:196] Checking for leaks (request via RPC)
07:56:53.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.815933 26653 ts_manager.cc:194] Re-registered known tserver with Master: bd0c2de15db9438f9df3c22ffa0bed6c (127.25.235.66:42989)
07:56:53.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.816005 26725 ts_manager.cc:194] Re-registered known tserver with Master: bd0c2de15db9438f9df3c22ffa0bed6c (127.25.235.66:42989)
07:56:53.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.818209 26725 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.235.66:51617
07:56:53.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.822338 28732 heartbeater.cc:499] Master 127.25.235.124:39273 was elected leader, sending a full tablet report...
07:56:53.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.920259 26541 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task2rU0iV/build/asan/bin/kudu with pid 28247
07:56:53.938 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
07:56:53.939 [DEBUG - kudu-nio-5] (Connection.java:239) [peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)] handling channelInactive
07:56:53.939 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)] cleaning up while in state READY due to: connection closed
07:56:53.959 [DEBUG - pool-2-thread-1] (Connection.java:748) [peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)] connecting to peer
07:56:53.961 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:53.961 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)] cleaning up while in state CONNECTING due to: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:53.961 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093) for tablet 9bc0a52dc27145398e9a7b4bdd4280b5: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:53.983 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:53.983 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671),953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)[L],bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)]}]
07:56:53.983 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, attempt=3, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=44), Traces: [0ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [0ms] delaying RPC due to: Illegal state: connection is terminated, [20ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [22ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [22ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [40ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [43ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=Deferred@2092772342(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f19483391e8@72c7f0d7 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@66a56a71 -> wakeup thread Time-limited test)) after lookup
07:56:53.984 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)] connecting to peer
07:56:53.985 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:53.985 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)] cleaning up while in state CONNECTING due to: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:53.985 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093) for tablet 9bc0a52dc27145398e9a7b4bdd4280b5: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:54.003 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:54.003 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671),953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)[L],bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)]}]
07:56:54.004 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, attempt=5, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=64), Traces: [0ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [0ms] delaying RPC due to: Illegal state: connection is terminated, [20ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [22ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [22ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [40ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [43ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [44ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [46ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [46ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [60ms] refreshing cache from master, [60ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [63ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=Deferred@2092772342(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f19483391e8@72c7f0d7 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@29897488 after retry RPC@1282619322), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@66a56a71 -> wakeup thread Time-limited test -> (continuation of Deferred@29897488 after retry RPC@1282619322))) after lookup
07:56:54.004 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)] connecting to peer
07:56:54.005 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:54.005 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)] cleaning up while in state CONNECTING due to: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:54.006 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093) for tablet 9bc0a52dc27145398e9a7b4bdd4280b5: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:54.082 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:54.082 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671),953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)[L],bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)]}]
07:56:54.083 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, attempt=7, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=143), Traces: [0ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [0ms] delaying RPC due to: Illegal state: connection is terminated, [20ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [22ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [22ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [40ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [43ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [44ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [46ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [46ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [60ms] refreshing cache from master, [60ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [63ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [65ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [67ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [67ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [140ms] refreshing cache from master, [140ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [143ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=Deferred@2092772342(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f19483391e8@72c7f0d7 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@29897488 after retry RPC@1282619322) -> (continuation of Deferred@581476176 after retry RPC@1999876696), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@66a56a71 -> wakeup thread Time-limited test -> (continuation of Deferred@29897488 after retry RPC@1282619322) -> (continuation of Deferred@581476176 after retry RPC@1999876696))) after lookup
07:56:54.083 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)] connecting to peer
07:56:54.084 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:54.084 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)] cleaning up while in state CONNECTING due to: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:54.084 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093) for tablet 9bc0a52dc27145398e9a7b4bdd4280b5: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:54.303 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:54.303 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671),953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)[L],bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)]}]
07:56:54.304 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, attempt=9, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=364), Traces: [0ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [0ms] delaying RPC due to: Illegal state: connection is terminated, [20ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [22ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [22ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [40ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [43ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [44ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [46ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [46ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [60ms] refreshing cache from master, [60ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [63ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [65ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [67ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [67ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [140ms] refreshing cache from master, [140ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [143ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [144ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [145ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [145ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [360ms] refreshing cache from master, [360ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [363ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=Deferred@2092772342(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f19483391e8@72c7f0d7 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@29897488 after retry RPC@1282619322) -> (continuation of Deferred@581476176 after retry RPC@1999876696) -> (continuation of Deferred@1919048961 after retry RPC@405377127), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@66a56a71 -> wakeup thread Time-limited test -> (continuation of Deferred@29897488 after retry RPC@1282619322) -> (continuation of Deferred@581476176 after retry RPC@1999876696) -> (continuation of Deferred@1919048961 after retry RPC@405377127))) after lookup
07:56:54.305 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)] connecting to peer
07:56:54.306 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:54.306 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)] cleaning up while in state CONNECTING due to: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:54.306 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093) for tablet 9bc0a52dc27145398e9a7b4bdd4280b5: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093
07:56:55.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.244539 28747 raft_consensus.cc:493] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 953ec1e7cb1345bd986b2b0171b6bcc2)
07:56:55.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.244728 28747 raft_consensus.cc:515] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:55.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.245576 28747 leader_election.cc:290] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 953ec1e7cb1345bd986b2b0171b6bcc2 (127.25.235.67:36093), bd0c2de15db9438f9df3c22ffa0bed6c (127.25.235.66:42989)
07:56:55.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:55.248116 28437 leader_election.cc:336] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 953ec1e7cb1345bd986b2b0171b6bcc2 (127.25.235.67:36093): Network error: Client connection negotiation failed: client connection to 127.25.235.67:36093: connect: Connection refused (error 111)
07:56:55.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.254944 28687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bc0a52dc27145398e9a7b4bdd4280b5" candidate_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" is_pre_election: true
07:56:55.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.255635 28687 raft_consensus.cc:2468] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3dd0f8e1dbda4e408e9caa288299ab5d in term 8.
07:56:55.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.256671 28436 leader_election.cc:304] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3dd0f8e1dbda4e408e9caa288299ab5d, bd0c2de15db9438f9df3c22ffa0bed6c; no voters: 953ec1e7cb1345bd986b2b0171b6bcc2
07:56:55.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.257108 28747 raft_consensus.cc:2804] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 9 FOLLOWER]: Leader pre-election won for term 10
07:56:55.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.257270 28747 raft_consensus.cc:493] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 9 FOLLOWER]: Starting leader election (detected failure of leader 953ec1e7cb1345bd986b2b0171b6bcc2)
07:56:55.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.257364 28747 raft_consensus.cc:3060] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 9 FOLLOWER]: Advancing to term 10
07:56:55.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.259613 28747 raft_consensus.cc:515] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:55.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.260298 28747 leader_election.cc:290] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [CANDIDATE]: Term 10 election: Requested vote from peers 953ec1e7cb1345bd986b2b0171b6bcc2 (127.25.235.67:36093), bd0c2de15db9438f9df3c22ffa0bed6c (127.25.235.66:42989)
07:56:55.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.260937 28687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bc0a52dc27145398e9a7b4bdd4280b5" candidate_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c"
07:56:55.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.261139 28687 raft_consensus.cc:3060] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [term 8 FOLLOWER]: Advancing to term 10
07:56:55.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:55.261952 28437 leader_election.cc:336] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 953ec1e7cb1345bd986b2b0171b6bcc2 (127.25.235.67:36093): Network error: Client connection negotiation failed: client connection to 127.25.235.67:36093: connect: Connection refused (error 111)
07:56:55.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.266361 28687 raft_consensus.cc:2468] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3dd0f8e1dbda4e408e9caa288299ab5d in term 10.
07:56:55.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.266836 28436 leader_election.cc:304] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3dd0f8e1dbda4e408e9caa288299ab5d, bd0c2de15db9438f9df3c22ffa0bed6c; no voters: 953ec1e7cb1345bd986b2b0171b6bcc2
07:56:55.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.267328 28747 raft_consensus.cc:2804] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 10 FOLLOWER]: Leader election won for term 10
07:56:55.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.272075 28747 raft_consensus.cc:697] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 10 LEADER]: Becoming Leader. State: Replica: 3dd0f8e1dbda4e408e9caa288299ab5d, State: Running, Role: LEADER
07:56:55.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.272665 28747 consensus_queue.cc:237] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 9.36, Last appended by leader: 36, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:55.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.276970 26725 catalog_manager.cc:5697] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d reported cstate change: term changed from 9 to 10, leader changed from 953ec1e7cb1345bd986b2b0171b6bcc2 (127.25.235.67) to 3dd0f8e1dbda4e408e9caa288299ab5d (127.25.235.65). New cstate: current_term: 10 leader_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } health_report { overall_health: UNKNOWN } } }
07:56:55.302 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:55.302 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)[L],953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093),bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)]}]
07:56:55.302 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, attempt=11, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=1363), Traces: [0ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [0ms] delaying RPC due to: Illegal state: connection is terminated, [20ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [22ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [22ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [40ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [43ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [44ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [46ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [46ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [60ms] refreshing cache from master, [60ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [63ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [65ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [67ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [67ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [140ms] refreshing cache from master, [140ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [143ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [144ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [145ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [145ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [360ms] refreshing cache from master, [360ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [363ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [365ms] sending RPC to server 953ec1e7cb1345bd986b2b0171b6bcc2, [367ms] received response from server 953ec1e7cb1345bd986b2b0171b6bcc2: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [367ms] delaying RPC due to: Network error: Failed to connect to peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093): Connection refused: /127.25.235.67:36093, [1359ms] refreshing cache from master, [1360ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [1362ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=Deferred@2092772342(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f19483391e8@72c7f0d7 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@29897488 after retry RPC@1282619322) -> (continuation of Deferred@581476176 after retry RPC@1999876696) -> (continuation of Deferred@1919048961 after retry RPC@405377127) -> (continuation of Deferred@1573202446 after retry RPC@1840715446), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@66a56a71 -> wakeup thread Time-limited test -> (continuation of Deferred@29897488 after retry RPC@1282619322) -> (continuation of Deferred@581476176 after retry RPC@1999876696) -> (continuation of Deferred@1919048961 after retry RPC@405377127) -> (continuation of Deferred@1573202446 after retry RPC@1840715446))) after lookup
07:56:55.303 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)] connecting to peer
07:56:55.304 [DEBUG - kudu-nio-3] (Connection.java:757) [peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)] Successfully connected to peer
07:56:55.319 [DEBUG - kudu-nio-3] (Negotiator.java:918) Authenticated connection [id: 0xe42142c3, L:/127.0.0.1:60016 - R:/127.25.235.65:44671] using TOKEN/null
07:56:55.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.326305 28687 raft_consensus.cc:1275] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [term 10 FOLLOWER]: Refusing update from remote peer 3dd0f8e1dbda4e408e9caa288299ab5d: Log matching property violated. Preceding OpId in replica: term: 8 index: 32. Preceding OpId from leader: term: 10 index: 38. (index mismatch)
07:56:55.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:55.326361 28437 consensus_peers.cc:597] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d -> Peer 953ec1e7cb1345bd986b2b0171b6bcc2 (127.25.235.67:36093): Couldn't send request to peer 953ec1e7cb1345bd986b2b0171b6bcc2. Status: Network error: Client connection negotiation failed: client connection to 127.25.235.67:36093: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
07:56:55.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.327250 28752 consensus_queue.cc:1048] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 32, Time since last communication: 0.000s
07:56:55.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.338341 28766 mvcc.cc:204] Tried to move back new op lower bound from 7305205413164187648 to 7305205412961779712. Current Snapshot: MvccSnapshot[applied={T|T < 7305205413164187648}]
07:56:55.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.340322 28768 mvcc.cc:204] Tried to move back new op lower bound from 7305205413164187648 to 7305205412961779712. Current Snapshot: MvccSnapshot[applied={T|T < 7305205413164187648}]
07:56:55.361 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.25.235.67:36093
07:56:55.361 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

07:56:55.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.362447 26541 external_mini_cluster.cc:1372] Running /tmp/dist-test-task2rU0iV/build/asan/bin/kudu
07:56:55.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task2rU0iV/build/asan/bin/kudu
07:56:55.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/wal
07:56:55.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/data
07:56:55.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
07:56:55.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
07:56:55.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
07:56:55.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:56:55.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:56:55.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:56:55.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/logs
07:56:55.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/data/info.pb
07:56:55.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:56:55.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
07:56:55.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
07:56:55.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
07:56:55.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
07:56:55.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:56:55.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:56:55.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:56:55.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
07:56:55.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
07:56:55.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.235.67:36093
07:56:55.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.25.235.67
07:56:55.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.25.235.126:46873,127.25.235.125:42159,127.25.235.124:39273
07:56:55.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=43721
07:56:55.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.235.67
07:56:55.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.235.84:44497
07:56:55.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:56:55.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:56:55.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
07:56:55.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:55.722949 28775 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
07:56:55.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:55.723424 28775 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
07:56:55.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:55.723555 28775 flags.cc:432] Enabled unsafe flag: --never_fsync=true
07:56:55.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:55.739168 28775 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
07:56:55.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:55.739439 28775 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.235.67
07:56:55.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.753921 28775 tablet_server_runner.cc:78] Tablet server non-default flags:
07:56:55.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:56:55.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.235.84:44497
07:56:55.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:56:55.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
07:56:55.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/data
07:56:55.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/wal
07:56:55.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:56:55.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:56:55.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.235.67:36093
07:56:55.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
07:56:55.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:56:55.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:56:55.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/data/info.pb
07:56:55.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.235.67
07:56:55.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=43721
07:56:55.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.25.235.126:46873,127.25.235.125:42159,127.25.235.124:39273
07:56:55.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
07:56:55.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:56:55.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
07:56:55.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
07:56:55.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:56:55.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.25.235.67
07:56:55.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/logs
07:56:55.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:56:55.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
07:56:55.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
07:56:55.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
07:56:55.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
07:56:55.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
07:56:55.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
07:56:55.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 07:43:22 UTC on bdcb31816ec0
07:56:55.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12770
07:56:55.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
07:56:55.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.755522 28775 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
07:56:55.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.757359 28775 file_cache.cc:504] Constructed file cache file cache with capacity 419430
07:56:55.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:55.767120 28780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:55.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:55.767069 28781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:55.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.768402 28775 server_base.cc:1061] running on GCE node
07:56:55.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:55.768594 28783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:55.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.769335 28775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
07:56:55.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.770431 28775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
07:56:55.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.771754 28775 hybrid_clock.cc:648] HybridClock initialized: now 1783497415771697 us; error 40 us; skew 500 ppm
07:56:55.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.772151 28775 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
07:56:55.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.774679 28775 webserver.cc:533] Webserver started at http://127.25.235.67:43721/ using document root <none> and password file <none>
07:56:55.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.775476 28775 fs_manager.cc:362] Metadata directory not provided
07:56:55.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.775611 28775 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
07:56:55.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.781303 28775 fs_manager.cc:714] Time spent opening directory manager: real 0.004s	user 0.002s	sys 0.001s
07:56:55.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.784555 28789 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
07:56:55.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.786247 28775 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
07:56:55.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.786396 28775 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/data,/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/wal
07:56:55.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "953ec1e7cb1345bd986b2b0171b6bcc2"
07:56:55.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 07:56:25 on dist-test-slave-63m1"
07:56:55.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.787226 28775 fs_report.cc:389] FS layout report
07:56:55.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
07:56:55.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/wal
07:56:55.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/wal
07:56:55.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/data/data
07:56:55.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
07:56:55.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
07:56:55.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
07:56:55.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
07:56:55.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
07:56:55.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
07:56:55.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
07:56:55.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
07:56:55.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
07:56:55.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
07:56:55.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
07:56:55.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.808797 28775 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
07:56:55.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.809715 28775 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
07:56:55.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.809967 28775 kserver.cc:163] Server-wide thread pool size limit: 3276
07:56:55.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.810876 28775 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
07:56:55.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.813144 28796 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
07:56:55.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.819249 28775 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
07:56:55.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.819401 28775 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.007s	user 0.001s	sys 0.000s
07:56:55.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.819591 28775 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
07:56:55.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.822925 28775 ts_tablet_manager.cc:616] Registered 1 tablets
07:56:55.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.823057 28775 ts_tablet_manager.cc:595] Time spent register tablets: real 0.004s	user 0.002s	sys 0.000s
07:56:55.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.823562 28796 tablet_bootstrap.cc:492] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2: Bootstrap starting.
07:56:55.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.853175 28796 log.cc:826] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2: Log is configured to *not* fsync() on all Append() calls
07:56:55.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.883571 28775 rpc_server.cc:307] RPC server started. Bound to: 127.25.235.67:36093
07:56:55.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.883633 28905 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.235.67:36093 every 8 connection(s)
07:56:55.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.885133 28775 server_base.cc:1193] Dumped server information to /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-2/data/info.pb
07:56:55.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.894150 26541 external_mini_cluster.cc:1434] Started /tmp/dist-test-task2rU0iV/build/asan/bin/kudu as pid 28775
07:56:55.894 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
07:56:55.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.904781 28796 tablet_bootstrap.cc:492] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=36 overwritten=0 applied=36 ignored=0} inserts{seen=27 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
07:56:55.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.905712 28796 tablet_bootstrap.cc:492] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2: Bootstrap complete.
07:56:55.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.907871 28796 ts_tablet_manager.cc:1403] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2: Time spent bootstrapping tablet: real 0.085s	user 0.056s	sys 0.018s
07:56:55.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.910266 28908 heartbeater.cc:344] Connected to a master server at 127.25.235.125:42159
07:56:55.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.910727 28908 heartbeater.cc:461] Registering TS with master...
07:56:55.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.911845 28908 heartbeater.cc:507] Master 127.25.235.125:42159 requested a full tablet report, sending...
07:56:55.915 [DEBUG - kudu-nio-3] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 30 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
07:56:55.916 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.25.235.65:44671
07:56:55.916 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

07:56:55.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.914017 28796 raft_consensus.cc:359] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [term 9 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:55.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.915069 28796 raft_consensus.cc:740] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [term 9 FOLLOWER]: Becoming Follower/Learner. State: Replica: 953ec1e7cb1345bd986b2b0171b6bcc2, State: Initialized, Role: FOLLOWER
07:56:55.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.915179 26653 ts_manager.cc:194] Re-registered known tserver with Master: 953ec1e7cb1345bd986b2b0171b6bcc2 (127.25.235.67:36093)
07:56:55.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.915793 28796 consensus_queue.cc:260] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 36, Last appended: 9.36, Last appended by leader: 36, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:55.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.918380 28796 ts_tablet_manager.cc:1434] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.005s
07:56:55.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.919185 26541 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-task2rU0iV/build/asan/bin/kudu pid 28420
07:56:55.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.919471 28906 heartbeater.cc:344] Connected to a master server at 127.25.235.124:39273
07:56:55.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.919611 28906 heartbeater.cc:461] Registering TS with master...
07:56:55.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.920192 28906 heartbeater.cc:507] Master 127.25.235.124:39273 requested a full tablet report, sending...
07:56:55.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.922453 28907 heartbeater.cc:344] Connected to a master server at 127.25.235.126:46873
07:56:55.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.922586 28907 heartbeater.cc:461] Registering TS with master...
07:56:55.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.922734 26725 ts_manager.cc:194] Re-registered known tserver with Master: 953ec1e7cb1345bd986b2b0171b6bcc2 (127.25.235.67:36093)
07:56:55.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.923128 28907 heartbeater.cc:507] Master 127.25.235.126:46873 requested a full tablet report, sending...
07:56:55.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.924160 26725 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.235.67:46295
07:56:55.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.924755 26583 ts_manager.cc:194] Re-registered known tserver with Master: 953ec1e7cb1345bd986b2b0171b6bcc2 (127.25.235.67:36093)
07:56:55.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.927847 28906 heartbeater.cc:499] Master 127.25.235.124:39273 was elected leader, sending a full tablet report...
07:56:55.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.937310 28544 generic_service.cc:196] Checking for leaks (request via RPC)
07:56:56.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:56.034451 26541 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task2rU0iV/build/asan/bin/kudu with pid 28420
07:56:56.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:56.053736 28616 connection.cc:573] server connection from 127.25.235.65:49233 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
07:56:56.054 [DEBUG - kudu-nio-3] (Connection.java:239) [peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)] handling channelInactive
07:56:56.054 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)] cleaning up while in state READY due to: connection closed
07:56:56.054 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
07:56:56.055 [DEBUG - Time-limited test] (Connection.java:748) [peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)] connecting to peer
07:56:56.056 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671
07:56:56.056 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671
07:56:56.056 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671) for tablet 9bc0a52dc27145398e9a7b4bdd4280b5: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671
07:56:56.061 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:56.061 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)[L],953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093),bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)]}]
07:56:56.062 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, attempt=2, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=7), Traces: [0ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [1ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [1ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [6ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=Deferred@1034202017(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f19483391e8@68730805 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6e3a0c90 -> wakeup thread Time-limited test)) after lookup
07:56:56.062 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)] connecting to peer
07:56:56.063 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671
07:56:56.063 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671
07:56:56.063 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671) for tablet 9bc0a52dc27145398e9a7b4bdd4280b5: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671
07:56:56.082 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:56.082 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)[L],953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093),bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)]}]
07:56:56.082 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, attempt=4, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=27), Traces: [0ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [1ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [1ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [6ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [7ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [8ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [8ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [27ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=Deferred@1034202017(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f19483391e8@68730805 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1683060148 after retry RPC@1521597218), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6e3a0c90 -> wakeup thread Time-limited test -> (continuation of Deferred@1683060148 after retry RPC@1521597218))) after lookup
07:56:56.083 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)] connecting to peer
07:56:56.084 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671
07:56:56.085 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671
07:56:56.085 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671) for tablet 9bc0a52dc27145398e9a7b4bdd4280b5: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671
07:56:56.122 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:56.122 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)[L],953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093),bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)]}]
07:56:56.122 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, attempt=6, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=67), Traces: [0ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [1ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [1ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [6ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [7ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [8ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [8ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [27ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [28ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [30ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [30ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [66ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=Deferred@1034202017(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f19483391e8@68730805 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1683060148 after retry RPC@1521597218) -> (continuation of Deferred@663588820 after retry RPC@1882343596), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6e3a0c90 -> wakeup thread Time-limited test -> (continuation of Deferred@1683060148 after retry RPC@1521597218) -> (continuation of Deferred@663588820 after retry RPC@1882343596))) after lookup
07:56:56.123 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)] connecting to peer
07:56:56.123 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671
07:56:56.124 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671
07:56:56.124 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671) for tablet 9bc0a52dc27145398e9a7b4bdd4280b5: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671
07:56:56.142 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:56.142 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)[L],953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093),bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)]}]
07:56:56.142 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, attempt=8, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=87), Traces: [0ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [1ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [1ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [6ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [7ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [8ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [8ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [27ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [28ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [30ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [30ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [66ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [67ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [69ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [69ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [83ms] refreshing cache from master, [83ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [86ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=Deferred@1034202017(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f19483391e8@68730805 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1683060148 after retry RPC@1521597218) -> (continuation of Deferred@663588820 after retry RPC@1882343596) -> (continuation of Deferred@371890151 after retry RPC@1950337702), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6e3a0c90 -> wakeup thread Time-limited test -> (continuation of Deferred@1683060148 after retry RPC@1521597218) -> (continuation of Deferred@663588820 after retry RPC@1882343596) -> (continuation of Deferred@371890151 after retry RPC@1950337702))) after lookup
07:56:56.143 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)] connecting to peer
07:56:56.144 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671
07:56:56.144 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671
07:56:56.144 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671) for tablet 9bc0a52dc27145398e9a7b4bdd4280b5: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671
07:56:56.362 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:56.362 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)[L],953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093),bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)]}]
07:56:56.362 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, attempt=10, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=307), Traces: [0ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [1ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [1ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [6ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [7ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [8ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [8ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [27ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [28ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [30ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [30ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [66ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [67ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [69ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [69ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [83ms] refreshing cache from master, [83ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [86ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [88ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [89ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [89ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [303ms] refreshing cache from master, [303ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [306ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=Deferred@1034202017(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f19483391e8@68730805 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1683060148 after retry RPC@1521597218) -> (continuation of Deferred@663588820 after retry RPC@1882343596) -> (continuation of Deferred@371890151 after retry RPC@1950337702) -> (continuation of Deferred@2021481538 after retry RPC@1922941713), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6e3a0c90 -> wakeup thread Time-limited test -> (continuation of Deferred@1683060148 after retry RPC@1521597218) -> (continuation of Deferred@663588820 after retry RPC@1882343596) -> (continuation of Deferred@371890151 after retry RPC@1950337702) -> (continuation of Deferred@2021481538 after retry RPC@1922941713))) after lookup
07:56:56.363 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)] connecting to peer
07:56:56.364 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671
07:56:56.364 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671
07:56:56.364 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671) for tablet 9bc0a52dc27145398e9a7b4bdd4280b5: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671
07:56:56.842 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:56.842 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)[L],953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093),bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)]}]
07:56:56.843 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, attempt=12, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=788), Traces: [0ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [1ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [1ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [6ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [7ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [8ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [8ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [27ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [28ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [30ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [30ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [66ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [67ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [69ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [69ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [83ms] refreshing cache from master, [83ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [86ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [88ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [89ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [89ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [303ms] refreshing cache from master, [303ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [306ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [308ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [309ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [309ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [784ms] refreshing cache from master, [784ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [787ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=Deferred@1034202017(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f19483391e8@68730805 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1683060148 after retry RPC@1521597218) -> (continuation of Deferred@663588820 after retry RPC@1882343596) -> (continuation of Deferred@371890151 after retry RPC@1950337702) -> (continuation of Deferred@2021481538 after retry RPC@1922941713) -> (continuation of Deferred@1779117036 after retry RPC@1989339383), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6e3a0c90 -> wakeup thread Time-limited test -> (continuation of Deferred@1683060148 after retry RPC@1521597218) -> (continuation of Deferred@663588820 after retry RPC@1882343596) -> (continuation of Deferred@371890151 after retry RPC@1950337702) -> (continuation of Deferred@2021481538 after retry RPC@1922941713) -> (continuation of Deferred@1779117036 after retry RPC@1989339383))) after lookup
07:56:56.843 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)] connecting to peer
07:56:56.844 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671
07:56:56.844 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671
07:56:56.844 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671) for tablet 9bc0a52dc27145398e9a7b4bdd4280b5: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671
07:56:57.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.377760 28921 raft_consensus.cc:493] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
07:56:57.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.378052 28921 raft_consensus.cc:515] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:57.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.379976 28921 leader_election.cc:290] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3dd0f8e1dbda4e408e9caa288299ab5d (127.25.235.65:44671), bd0c2de15db9438f9df3c22ffa0bed6c (127.25.235.66:42989)
07:56:57.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:57.381594 28791 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.235.65:44671: connect: Connection refused (error 111)
07:56:57.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:57.385326 28791 leader_election.cc:336] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 3dd0f8e1dbda4e408e9caa288299ab5d (127.25.235.65:44671): Network error: Client connection negotiation failed: client connection to 127.25.235.65:44671: connect: Connection refused (error 111)
07:56:57.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.388351 28687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bc0a52dc27145398e9a7b4bdd4280b5" candidate_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" is_pre_election: true
07:56:57.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.389238 28791 leader_election.cc:304] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 953ec1e7cb1345bd986b2b0171b6bcc2; no voters: 3dd0f8e1dbda4e408e9caa288299ab5d, bd0c2de15db9438f9df3c22ffa0bed6c
07:56:57.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.389635 28921 raft_consensus.cc:2749] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
07:56:57.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.536403 28926 raft_consensus.cc:493] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 3dd0f8e1dbda4e408e9caa288299ab5d)
07:56:57.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.536701 28926 raft_consensus.cc:515] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:57.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.538414 28926 leader_election.cc:290] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 953ec1e7cb1345bd986b2b0171b6bcc2 (127.25.235.67:36093), 3dd0f8e1dbda4e408e9caa288299ab5d (127.25.235.65:44671)
07:56:57.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:57.539814 28617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.235.65:44671: connect: Connection refused (error 111)
07:56:57.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:57.541646 28617 leader_election.cc:336] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 3dd0f8e1dbda4e408e9caa288299ab5d (127.25.235.65:44671): Network error: Client connection negotiation failed: client connection to 127.25.235.65:44671: connect: Connection refused (error 111)
07:56:57.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.547716 28861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bc0a52dc27145398e9a7b4bdd4280b5" candidate_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" candidate_term: 11 candidate_status { last_received { term: 10 index: 40 } } ignore_live_leader: false dest_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" is_pre_election: true
07:56:57.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.548197 28861 raft_consensus.cc:2468] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd0c2de15db9438f9df3c22ffa0bed6c in term 9.
07:56:57.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.549474 28618 leader_election.cc:304] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 953ec1e7cb1345bd986b2b0171b6bcc2, bd0c2de15db9438f9df3c22ffa0bed6c; no voters: 3dd0f8e1dbda4e408e9caa288299ab5d
07:56:57.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.549958 28926 raft_consensus.cc:2804] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [term 10 FOLLOWER]: Leader pre-election won for term 11
07:56:57.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.550107 28926 raft_consensus.cc:493] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [term 10 FOLLOWER]: Starting leader election (detected failure of leader 3dd0f8e1dbda4e408e9caa288299ab5d)
07:56:57.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.550191 28926 raft_consensus.cc:3060] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [term 10 FOLLOWER]: Advancing to term 11
07:56:57.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.552386 28926 raft_consensus.cc:515] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:57.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.553098 28926 leader_election.cc:290] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [CANDIDATE]: Term 11 election: Requested vote from peers 953ec1e7cb1345bd986b2b0171b6bcc2 (127.25.235.67:36093), 3dd0f8e1dbda4e408e9caa288299ab5d (127.25.235.65:44671)
07:56:57.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.553763 28861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bc0a52dc27145398e9a7b4bdd4280b5" candidate_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" candidate_term: 11 candidate_status { last_received { term: 10 index: 40 } } ignore_live_leader: false dest_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2"
07:56:57.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.553984 28861 raft_consensus.cc:3060] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [term 9 FOLLOWER]: Advancing to term 11
07:56:57.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:57.556815 28617 leader_election.cc:336] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 3dd0f8e1dbda4e408e9caa288299ab5d (127.25.235.65:44671): Network error: Client connection negotiation failed: client connection to 127.25.235.65:44671: connect: Connection refused (error 111)
07:56:57.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.558036 28861 raft_consensus.cc:2468] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd0c2de15db9438f9df3c22ffa0bed6c in term 11.
07:56:57.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.558534 28618 leader_election.cc:304] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 953ec1e7cb1345bd986b2b0171b6bcc2, bd0c2de15db9438f9df3c22ffa0bed6c; no voters: 3dd0f8e1dbda4e408e9caa288299ab5d
07:56:57.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.559077 28926 raft_consensus.cc:2804] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [term 11 FOLLOWER]: Leader election won for term 11
07:56:57.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.559602 28926 raft_consensus.cc:697] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [term 11 LEADER]: Becoming Leader. State: Replica: bd0c2de15db9438f9df3c22ffa0bed6c, State: Running, Role: LEADER
07:56:57.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.560246 28926 consensus_queue.cc:237] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 10.40, Last appended by leader: 40, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:57.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.564409 26725 catalog_manager.cc:5697] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c reported cstate change: term changed from 10 to 11, leader changed from 3dd0f8e1dbda4e408e9caa288299ab5d (127.25.235.65) to bd0c2de15db9438f9df3c22ffa0bed6c (127.25.235.66). New cstate: current_term: 11 leader_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } health_report { overall_health: HEALTHY } } }
07:56:58.002 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 9bc0a52dc27145398e9a7b4bdd4280b5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721' with partition [<start>, <end>)
07:56:58.003 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=9bc0a52dc27145398e9a7b4bdd4280b5@[3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671),953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093),bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)[L]]}]
07:56:58.003 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, attempt=14, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=1948), Traces: [0ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [1ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [1ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [6ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [7ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [8ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [8ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [27ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [28ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [30ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [30ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [66ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [67ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [69ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [69ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [83ms] refreshing cache from master, [83ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [86ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [88ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [89ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [89ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [303ms] refreshing cache from master, [303ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [306ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [308ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [309ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [309ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [784ms] refreshing cache from master, [784ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [787ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, [788ms] sending RPC to server 3dd0f8e1dbda4e408e9caa288299ab5d, [789ms] received response from server 3dd0f8e1dbda4e408e9caa288299ab5d: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [789ms] delaying RPC due to: Network error: Failed to connect to peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671): Connection refused: /127.25.235.65:44671, [1944ms] refreshing cache from master, [1944ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.235.124:39273, [1947ms] Sub RPC GetTableLocations: received response from server master-127.25.235.124:39273: OK, deferred=Deferred@1034202017(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f19483391e8@68730805 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1683060148 after retry RPC@1521597218) -> (continuation of Deferred@663588820 after retry RPC@1882343596) -> (continuation of Deferred@371890151 after retry RPC@1950337702) -> (continuation of Deferred@2021481538 after retry RPC@1922941713) -> (continuation of Deferred@1779117036 after retry RPC@1989339383) -> (continuation of Deferred@247925568 after retry RPC@2098304899), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6e3a0c90 -> wakeup thread Time-limited test -> (continuation of Deferred@1683060148 after retry RPC@1521597218) -> (continuation of Deferred@663588820 after retry RPC@1882343596) -> (continuation of Deferred@371890151 after retry RPC@1950337702) -> (continuation of Deferred@2021481538 after retry RPC@1922941713) -> (continuation of Deferred@1779117036 after retry RPC@1989339383) -> (continuation of Deferred@247925568 after retry RPC@2098304899))) after lookup
07:56:58.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.004448 28861 raft_consensus.cc:1275] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 953ec1e7cb1345bd986b2b0171b6bcc2 [term 11 FOLLOWER]: Refusing update from remote peer bd0c2de15db9438f9df3c22ffa0bed6c: Log matching property violated. Preceding OpId in replica: term: 9 index: 36. Preceding OpId from leader: term: 11 index: 41. (index mismatch)
07:56:58.005 [DEBUG - kudu-nio-2] (Connection.java:748) [peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)] connecting to peer
07:56:58.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.005529 28926 consensus_queue.cc:1048] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 36, Time since last communication: 0.000s
07:56:58.006 [DEBUG - kudu-nio-3] (Connection.java:757) [peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)] Successfully connected to peer
07:56:58.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.014889 28946 mvcc.cc:204] Tried to move back new op lower bound from 7305205413164187648 to 7305205412961779712. Current Snapshot: MvccSnapshot[applied={T|T < 7305205413164187648}]
07:56:58.029 [DEBUG - kudu-nio-3] (Negotiator.java:918) Authenticated connection [id: 0x2dc29a96, L:/127.0.0.1:44576 - R:/127.25.235.66:42989] using TOKEN/null
07:56:58.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:58.039654 28617 consensus_peers.cc:597] T 9bc0a52dc27145398e9a7b4bdd4280b5 P bd0c2de15db9438f9df3c22ffa0bed6c -> Peer 3dd0f8e1dbda4e408e9caa288299ab5d (127.25.235.65:44671): Couldn't send request to peer 3dd0f8e1dbda4e408e9caa288299ab5d. Status: Network error: Client connection negotiation failed: client connection to 127.25.235.65:44671: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
07:56:58.060 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.25.235.65:44671
07:56:58.060 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

07:56:58.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.061170 26541 external_mini_cluster.cc:1372] Running /tmp/dist-test-task2rU0iV/build/asan/bin/kudu
07:56:58.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task2rU0iV/build/asan/bin/kudu
07:56:58.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-0/wal
07:56:58.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-0/data
07:56:58.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
07:56:58.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
07:56:58.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
07:56:58.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:56:58.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:56:58.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:56:58.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-0/logs
07:56:58.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-0/data/info.pb
07:56:58.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:56:58.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
07:56:58.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
07:56:58.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
07:56:58.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
07:56:58.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:56:58.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:56:58.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:56:58.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
07:56:58.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
07:56:58.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.235.65:44671
07:56:58.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.25.235.65
07:56:58.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.25.235.126:46873,127.25.235.125:42159,127.25.235.124:39273
07:56:58.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42151
07:56:58.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.235.65
07:56:58.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.235.84:44497
07:56:58.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:56:58.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:56:58.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
07:56:58.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:58.415782 28953 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
07:56:58.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:58.416107 28953 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
07:56:58.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:58.416191 28953 flags.cc:432] Enabled unsafe flag: --never_fsync=true
07:56:58.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:58.425107 28953 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
07:56:58.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:58.425280 28953 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.235.65
07:56:58.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.435992 28953 tablet_server_runner.cc:78] Tablet server non-default flags:
07:56:58.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:56:58.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.235.84:44497
07:56:58.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:56:58.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
07:56:58.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-0/data
07:56:58.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-0/wal
07:56:58.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:56:58.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:56:58.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.235.65:44671
07:56:58.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
07:56:58.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:56:58.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:56:58.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-0/data/info.pb
07:56:58.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.235.65
07:56:58.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42151
07:56:58.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.25.235.126:46873,127.25.235.125:42159,127.25.235.124:39273
07:56:58.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
07:56:58.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:56:58.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
07:56:58.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
07:56:58.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:56:58.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.25.235.65
07:56:58.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-0/logs
07:56:58.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:56:58.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
07:56:58.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
07:56:58.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
07:56:58.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
07:56:58.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
07:56:58.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
07:56:58.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 07:43:22 UTC on bdcb31816ec0
07:56:58.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12770
07:56:58.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
07:56:58.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.437702 28953 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
07:56:58.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.439496 28953 file_cache.cc:504] Constructed file cache file cache with capacity 419430
07:56:58.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:58.449355 28959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:58.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:58.450047 28961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:58.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:58.450196 28958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:58.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.450541 28953 server_base.cc:1061] running on GCE node
07:56:58.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.451197 28953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
07:56:58.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.452277 28953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
07:56:58.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.453544 28953 hybrid_clock.cc:648] HybridClock initialized: now 1783497418453425 us; error 102 us; skew 500 ppm
07:56:58.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.453922 28953 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
07:56:58.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.456296 28953 webserver.cc:533] Webserver started at http://127.25.235.65:42151/ using document root <none> and password file <none>
07:56:58.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.457193 28953 fs_manager.cc:362] Metadata directory not provided
07:56:58.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.457329 28953 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
07:56:58.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.463299 28953 fs_manager.cc:714] Time spent opening directory manager: real 0.004s	user 0.005s	sys 0.000s
07:56:58.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.466605 28967 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
07:56:58.468 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.468307 28953 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.003s
07:56:58.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.468535 28953 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-0/data,/tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-0/wal
07:56:58.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "3dd0f8e1dbda4e408e9caa288299ab5d"
07:56:58.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 07:56:24 on dist-test-slave-63m1"
07:56:58.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.469532 28953 fs_report.cc:389] FS layout report
07:56:58.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
07:56:58.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-0/wal
07:56:58.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-0/wal
07:56:58.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-0/data/data
07:56:58.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
07:56:58.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
07:56:58.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
07:56:58.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
07:56:58.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
07:56:58.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
07:56:58.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
07:56:58.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
07:56:58.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
07:56:58.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
07:56:58.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
07:56:58.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.505378 28953 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
07:56:58.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.506420 28953 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
07:56:58.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.506825 28953 kserver.cc:163] Server-wide thread pool size limit: 3276
07:56:58.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.508002 28953 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
07:56:58.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.510501 28974 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
07:56:58.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.516845 28953 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
07:56:58.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.517040 28953 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.007s	user 0.000s	sys 0.001s
07:56:58.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.517192 28953 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
07:56:58.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.520438 28953 ts_tablet_manager.cc:616] Registered 1 tablets
07:56:58.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.520622 28953 ts_tablet_manager.cc:595] Time spent register tablets: real 0.003s	user 0.000s	sys 0.000s
07:56:58.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.521044 28974 tablet_bootstrap.cc:492] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d: Bootstrap starting.
07:56:58.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.550551 28974 log.cc:826] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d: Log is configured to *not* fsync() on all Append() calls
07:56:58.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.571350 28953 rpc_server.cc:307] RPC server started. Bound to: 127.25.235.65:44671
07:56:58.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.571405 29081 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.235.65:44671 every 8 connection(s)
07:56:58.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.573115 28953 server_base.cc:1193] Dumped server information to /tmp/dist-test-task2rU0iV/test-tmp/mini-kudu-cluster11788247458870820629/ts-0/data/info.pb
07:56:58.574 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
07:56:58.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.573946 26541 external_mini_cluster.cc:1434] Started /tmp/dist-test-task2rU0iV/build/asan/bin/kudu as pid 28953
07:56:58.599 [DEBUG - kudu-nio-3] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 33 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
07:56:58.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.601100 28974 tablet_bootstrap.cc:492] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d: Bootstrap replayed 1/1 log segments. Stats: ops{read=40 overwritten=0 applied=40 ignored=0} inserts{seen=30 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
07:56:58.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.601940 28974 tablet_bootstrap.cc:492] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d: Bootstrap complete.
07:56:58.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.603472 28974 ts_tablet_manager.cc:1403] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d: Time spent bootstrapping tablet: real 0.083s	user 0.065s	sys 0.012s
07:56:58.607 [DEBUG - kudu-nio-3] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 33 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783497385721, tablet=9bc0a52dc27145398e9a7b4bdd4280b5, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
07:56:58.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.611812 28974 raft_consensus.cc:359] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 10 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:58.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.612856 28974 raft_consensus.cc:740] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [term 10 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dd0f8e1dbda4e408e9caa288299ab5d, State: Initialized, Role: FOLLOWER
07:56:58.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.613749 28974 consensus_queue.cc:260] T 9bc0a52dc27145398e9a7b4bdd4280b5 P 3dd0f8e1dbda4e408e9caa288299ab5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 40, Last appended: 10.40, Last appended by leader: 40, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "953ec1e7cb1345bd986b2b0171b6bcc2" member_type: VOTER last_known_addr { host: "127.25.235.67" port: 36093 } } peers { permanent_uuid: "3dd0f8e1dbda4e408e9caa288299ab5d" member_type: VOTER last_known_addr { host: "127.25.235.65" port: 44671 } } peers { permanent_uuid: "bd0c2de15db9438f9df3c22ffa0bed6c" member_type: VOTER last_known_addr { host: "127.25.235.66" port: 42989 } }
07:56:58.615 [DEBUG - main] (Connection.java:575) [peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)] disconnecting while in state READY
07:56:58.620 [DEBUG - kudu-nio-3] (Connection.java:239) [peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)] handling channelInactive
07:56:58.621 [DEBUG - main] (Connection.java:575) [peer master-127.25.235.125:42159(127.25.235.125:42159)] disconnecting while in state READY
07:56:58.621 [DEBUG - main] (Connection.java:575) [peer 953ec1e7cb1345bd986b2b0171b6bcc2(127.25.235.67:36093)] disconnecting while in state TERMINATED
07:56:58.621 [DEBUG - main] (Connection.java:575) [peer 3dd0f8e1dbda4e408e9caa288299ab5d(127.25.235.65:44671)] disconnecting while in state TERMINATED
07:56:58.621 [DEBUG - main] (Connection.java:575) [peer master-127.25.235.126:46873(127.25.235.126:46873)] disconnecting while in state READY
07:56:58.622 [DEBUG - kudu-nio-3] (Connection.java:709) [peer bd0c2de15db9438f9df3c22ffa0bed6c(127.25.235.66:42989)] cleaning up while in state READY due to: connection closed
07:56:58.622 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.25.235.126:46873(127.25.235.126:46873)] handling channelInactive
07:56:58.623 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.25.235.126:46873(127.25.235.126:46873)] cleaning up while in state READY due to: connection closed
07:56:58.623 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.25.235.125:42159(127.25.235.125:42159)] handling channelInactive
07:56:58.623 [DEBUG - main] (Connection.java:575) [peer master-127.25.235.124:39273(127.25.235.124:39273)] disconnecting while in state READY
07:56:58.623 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.25.235.125:42159(127.25.235.125:42159)] cleaning up while in state READY due to: connection closed
07:56:58.624 [DEBUG - kudu-nio-2] (Connection.java:239) [peer master-127.25.235.124:39273(127.25.235.124:39273)] handling channelInactive
07:56:58.624 [DEBUG - main] (AsyncKuduClient.java:2704) Releasing all remaining resources
07:56:58.624 [DEBUG - kudu-nio-2] (Connection.java:709) [peer master-127.25.235.124:39273(127.25.235.124:39273)] cleaning up while in state READY due to: connection closed
07:56:58.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.627753 26541 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-task2rU0iV/build/asan/bin/kudu pid 28953
07:56:58.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.629940 29083 heartbeater.cc:344] Connected to a master server at 127.25.235.126:46873
07:56:58.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.630396 29083 heartbeater.cc:461] Registering TS with master...
07:56:58.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.631700 29083 heartbeater.cc:507] Master 127.25.235.126:46873 requested a full tablet report, sending...
07:56:58.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.636598 26583 ts_manager.cc:194] Re-registered known tserver with Master: 3dd0f8e1dbda4e408e9caa288299ab5d (127.25.235.65:44671)
07:56:58.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.659245 29082 heartbeater.cc:344] Connected to a master server at 127.25.235.124:39273
07:56:58.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.659303 29089 heartbeater.cc:344] Connected to a master server at 127.25.235.125:42159
07:56:58.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.659545 29082 heartbeater.cc:461] Registering TS with master...
07:56:58.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.659548 29089 heartbeater.cc:461] Registering TS with master...
07:56:58.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.660313 29089 heartbeater.cc:507] Master 127.25.235.125:42159 requested a full tablet report, sending...
07:56:58.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.661278 29082 heartbeater.cc:507] Master 127.25.235.124:39273 requested a full tablet report, sending...
07:56:58.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.665243 29077 generic_service.cc:196] Checking for leaks (request via RPC)
07:56:58.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.665609 26725 ts_manager.cc:194] Re-registered known tserver with Master: 3dd0f8e1dbda4e408e9caa288299ab5d (127.25.235.65:44671)
07:56:58.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.667205 26725 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.235.65:46905
07:56:58.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.669389 26653 ts_manager.cc:194] Re-registered known tserver with Master: 3dd0f8e1dbda4e408e9caa288299ab5d (127.25.235.65:44671)
07:56:58.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.767376 26541 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task2rU0iV/build/asan/bin/kudu with pid 28953
07:56:58.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.790901 26541 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-task2rU0iV/build/asan/bin/kudu pid 28601
07:56:58.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:58.791643 29098 negotiation.cc:336] Failed RPC negotiation. Trace:
07:56:58.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 07:56:58.641089 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.25.235.65:44671 (local address 127.25.235.66:58865)
07:56:58.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 07:56:58.656618 (+ 15529us) negotiation.cc:107] Waiting for socket to connect
07:56:58.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 07:56:58.656638 (+    20us) client_negotiation.cc:175] Beginning negotiation
07:56:58.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 07:56:58.656792 (+   154us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
07:56:58.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 07:56:58.661929 (+  5137us) client_negotiation.cc:281] Received NEGOTIATE NegotiatePB response
07:56:58.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 07:56:58.661936 (+     7us) client_negotiation.cc:404] Received NEGOTIATE response from server
07:56:58.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 07:56:58.661971 (+    35us) client_negotiation.cc:190] Negotiated authn=SASL
07:56:58.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 07:56:58.662619 (+   648us) client_negotiation.cc:530] Sending TLS_HANDSHAKE message to server
07:56:58.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 07:56:58.662637 (+    18us) client_negotiation.cc:262] Sending TLS_HANDSHAKE NegotiatePB request
07:56:58.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 07:56:58.766130 (+103493us) client_negotiation.cc:281] Received TLS_HANDSHAKE NegotiatePB response
07:56:58.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 07:56:58.766137 (+     7us) client_negotiation.cc:543] Received TLS_HANDSHAKE response from server
07:56:58.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 07:56:58.767379 (+  1242us) client_negotiation.cc:530] Sending TLS_HANDSHAKE message to server
07:56:58.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 07:56:58.767400 (+    21us) client_negotiation.cc:262] Sending TLS_HANDSHAKE NegotiatePB request
07:56:58.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 07:56:58.768215 (+   815us) client_negotiation.cc:564] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
07:56:58.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 07:56:58.769032 (+   817us) client_negotiation.cc:639] Initiating SASL PLAIN handshake
07:56:58.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 07:56:58.769039 (+     7us) client_negotiation.cc:672] calling sasl_client_start()
07:56:58.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 07:56:58.769064 (+    25us) client_negotiation.cc:830] callback for SASL_CB_AUTHNAME
07:56:58.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 07:56:58.769121 (+    57us) client_negotiation.cc:262] Sending SASL_INITIATE NegotiatePB request
07:56:58.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 07:56:58.790313 (+ 21192us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.235.65:44671: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
07:56:58.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"client-negotiator.queue_time_us":15400,"thread_start_us":15783,"threads_started":1}
07:56:58.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.800987 28727 generic_service.cc:196] Checking for leaks (request via RPC)
07:56:58.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.921468 26541 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task2rU0iV/build/asan/bin/kudu with pid 28601
07:56:58.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:58.940366 28790 connection.cc:573] server connection from 127.25.235.66:41097 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
07:56:58.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.940688 26541 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-task2rU0iV/build/asan/bin/kudu pid 28775
07:56:58.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.947824 28901 generic_service.cc:196] Checking for leaks (request via RPC)
07:56:59.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.040598 26541 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task2rU0iV/build/asan/bin/kudu with pid 28775
07:56:59.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:59.057827 26641 connection.cc:573] server connection from 127.25.235.67:46603 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
07:56:59.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:59.057933 26711 connection.cc:573] server connection from 127.25.235.67:46295 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
07:56:59.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:59.057900 26570 connection.cc:573] server connection from 127.25.235.67:36573 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
07:56:59.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.058539 26541 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-task2rU0iV/build/asan/bin/kudu pid 26552
07:56:59.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.059856 26613 generic_service.cc:196] Checking for leaks (request via RPC)
07:56:59.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.175238 26541 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task2rU0iV/build/asan/bin/kudu with pid 26552
07:56:59.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.197983 26541 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-task2rU0iV/build/asan/bin/kudu pid 26620
07:56:59.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.199322 26684 generic_service.cc:196] Checking for leaks (request via RPC)
07:56:59.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.311517 26541 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task2rU0iV/build/asan/bin/kudu with pid 26620
07:56:59.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:59.333245 26712 connection.cc:573] client connection to 127.25.235.125:42159 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
07:56:59.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:59.333541 26712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
07:56:59.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.333751 26541 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-task2rU0iV/build/asan/bin/kudu pid 26691
07:56:59.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.335093 26755 generic_service.cc:196] Checking for leaks (request via RPC)
07:56:59.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:59.452780 26712 consensus_peers.cc:597] T 00000000000000000000000000000000 P c100d5f546314da4a0df1ef31e8f4b2c -> Peer 947b6ed1d6bb49e49d50a03e028cbd24 (127.25.235.125:42159): Couldn't send request to peer 947b6ed1d6bb49e49d50a03e028cbd24. Status: Network error: Client connection negotiation failed: client connection to 127.25.235.125:42159: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
07:56:59.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.453023 26541 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task2rU0iV/build/asan/bin/kudu with pid 26691
07:56:59.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-07-08T07:56:59Z chronyd exiting

Time: 39.049

OK (1 test)

