JUnit version 4.13.2
.02:06:56.360 [DEBUG - main] (RandomUtils.java:49) Using random seed: 1783476416351
02:06:56.423 [INFO - main] (RetryRule.java:138) Creating RetryStatement with result reporter and retry count of 0 (no retries)
02:06:56.490 [INFO - main] (KuduTestHarness.java:155) Creating a new MiniKuduCluster...
02:06:56.501 [INFO - main] (TempDirUtils.java:71) Using the temp directory defined by TEST_TMPDIR: /tmp/dist-test-taskQUnVk7/test-tmp
02:06:56.506 [INFO - main] (KuduBinaryLocator.java:83) Using Kudu binary directory specified by system property 'kuduBinDir': ../../../build/asan/bin
02:06:56.508 [INFO - main] (MiniKuduCluster.java:217) Starting process: [/tmp/dist-test-taskQUnVk7/java/build/dist-test/../../../build/asan/bin/kudu, test, mini_cluster, --serialization=pb]
02:06:56.692 [DEBUG - main] (MiniKuduCluster.java:185) Request: create_cluster {
  num_masters: 3
  num_tservers: 3
  enable_kerberos: false
  cluster_root: "/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309"
  hms_mode: NONE
  mini_kdc_options {
  }
  principal: "kudu"
  mini_oidc_options {
  }
}

02:06:56.860 [DEBUG - main] (MiniKuduCluster.java:195) Response: 
02:06:56.861 [DEBUG - main] (MiniKuduCluster.java:185) Request: start_cluster {
}

02:06:56.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:56.864696 21561 mini_oidc.cc:150] Starting JWKS server
02:06:56.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:56.866932 21561 webserver.cc:533] Webserver started at http://127.0.0.1:39027/ using document root <none> and password file <none>
02:06:56.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:56.867159 21561 mini_oidc.cc:186] Starting OIDC Discovery server
02:06:56.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:56.868070 21561 webserver.cc:533] Webserver started at http://127.0.0.1:39869/ using document root <none> and password file <none>
02:06:56.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-07-08T02:06:56Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
02:06:56.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-07-08T02:06:56Z Disabled control of system clock
02:06:56.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:56.900358 21561 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu
02:06:56.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu
02:06:56.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/master-0/wal
02:06:56.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/master-0/data
02:06:56.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:06:56.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:06:56.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:06:56.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:06:56.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:06:56.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:06:56.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/master-0/logs
02:06:56.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/master-0/data/info.pb
02:06:56.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:06:56.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:06:56.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:06:56.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:06:56.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:06:56.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:06:56.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:06:56.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:06:56.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
02:06:56.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:06:56.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
02:06:56.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
02:06:56.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.14.126:34291
02:06:56.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.14.126
02:06:56.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:06:56.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.14.84:44189
02:06:56.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:06:56.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:06:56.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:06:56.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
02:06:56.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.21.14.126:34291,127.21.14.125:33787,127.21.14.124:36603
02:06:56.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
02:06:56.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:39027/jwks.json with env {}
02:06:57.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:57.212628 21572 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:06:57.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:57.212901 21572 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:06:57.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:57.212977 21572 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:06:57.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:57.221374 21572 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
02:06:57.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:57.221447 21572 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:06:57.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:57.221500 21572 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
02:06:57.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:57.221556 21572 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
02:06:57.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:57.221624 21572 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
02:06:57.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:57.221670 21572 flags.cc:432] Enabled experimental flag: --jwks_url=http://localhost:39027/jwks.json
02:06:57.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.231724 21572 master_runner.cc:387] Master server non-default flags:
02:06:57.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:06:57.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.14.84:44189
02:06:57.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:06:57.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:06:57.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/master-0/data
02:06:57.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/master-0/wal
02:06:57.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
02:06:57.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.21.14.126:34291,127.21.14.125:33787,127.21.14.124:36603
02:06:57.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:06:57.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:06:57.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
02:06:57.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.14.126:34291
02:06:57.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
02:06:57.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:06:57.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
02:06:57.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:39027/jwks.json
02:06:57.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:06:57.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:06:57.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/master-0/data/info.pb
02:06:57.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.14.126
02:06:57.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:06:57.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:06:57.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:06:57.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:06:57.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:06:57.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:06:57.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/master-0/logs
02:06:57.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:06:57.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:06:57.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:06:57.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
02:06:57.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:06:57.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
02:06:57.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:06:57.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:43:30 UTC on bdcb31816ec0
02:06:57.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12766
02:06:57.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
02:06:57.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.233687 21572 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:06:57.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.235841 21572 file_cache.cc:504] Constructed file cache file cache with capacity 419430
02:06:57.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:57.244212 21580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:06:57.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:57.244359 21577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:06:57.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:57.244277 21578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:06:57.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.245743 21572 server_base.cc:1061] running on GCE node
02:06:57.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.246472 21572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:06:57.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.247865 21572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:06:57.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.249053 21572 hybrid_clock.cc:648] HybridClock initialized: now 1783476417248997 us; error 48 us; skew 500 ppm
02:06:57.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.249354 21572 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:06:57.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.251242 21572 webserver.cc:533] Webserver started at http://127.21.14.126:36679/ using document root <none> and password file <none>
02:06:57.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.251858 21572 fs_manager.cc:362] Metadata directory not provided
02:06:57.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.251938 21572 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:06:57.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.252264 21572 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
02:06:57.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.254420 21572 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/master-0/data/instance:
02:06:57.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "492faeffff9b4345a5b964c4d6f2b7e6"
02:06:57.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:06:57 on dist-test-slave-545r"
02:06:57.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.255129 21572 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/master-0/wal/instance:
02:06:57.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "492faeffff9b4345a5b964c4d6f2b7e6"
02:06:57.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:06:57 on dist-test-slave-545r"
02:06:57.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.260012 21572 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.002s
02:06:57.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.263095 21586 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:06:57.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.264438 21572 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
02:06:57.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.264604 21572 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/master-0/data,/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/master-0/wal
02:06:57.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "492faeffff9b4345a5b964c4d6f2b7e6"
02:06:57.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:06:57 on dist-test-slave-545r"
02:06:57.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.264806 21572 fs_report.cc:389] FS layout report
02:06:57.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:06:57.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/master-0/wal
02:06:57.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/master-0/wal
02:06:57.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/master-0/data/data
02:06:57.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:06:57.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:06:57.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:06:57.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:06:57.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:06:57.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:06:57.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:06:57.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:06:57.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:06:57.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:06:57.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:06:57.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.281513 21572 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:06:57.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.282238 21572 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:06:57.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.282442 21572 kserver.cc:163] Server-wide thread pool size limit: 3276
02:06:57.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.299990 21572 rpc_server.cc:307] RPC server started. Bound to: 127.21.14.126:34291
02:06:57.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.300009 21637 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.14.126:34291 every 8 connection(s)
02:06:57.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.301389 21572 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/master-0/data/info.pb
02:06:57.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.305372 21638 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
02:06:57.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.305544 21561 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu as pid 21572
02:06:57.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.305828 21561 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/master-0/wal/instance
02:06:57.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.307971 21561 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu
02:06:57.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu
02:06:57.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/master-1/wal
02:06:57.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/master-1/data
02:06:57.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:06:57.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:06:57.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:06:57.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:06:57.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:06:57.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:06:57.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/master-1/logs
02:06:57.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/master-1/data/info.pb
02:06:57.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:06:57.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:06:57.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:06:57.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:06:57.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:06:57.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:06:57.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:06:57.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:06:57.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
02:06:57.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:06:57.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
02:06:57.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
02:06:57.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.14.125:33787
02:06:57.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.14.125
02:06:57.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:06:57.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.14.84:44189
02:06:57.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:06:57.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:06:57.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:06:57.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
02:06:57.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.21.14.126:34291,127.21.14.125:33787,127.21.14.124:36603
02:06:57.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
02:06:57.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:39027/jwks.json with env {}
02:06:57.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.312734 21638 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.14.126" port: 34291 } has no permanent_uuid. Determining permanent_uuid...
02:06:57.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.328506 21638 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.14.125" port: 33787 } has no permanent_uuid. Determining permanent_uuid...
02:06:57.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:57.329409 21589 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.14.125:33787: connect: Connection refused (error 111)
02:06:57.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:57.330781 21638 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.21.14.125:33787: Network error: Client connection negotiation failed: client connection to 127.21.14.125:33787: connect: Connection refused (error 111)
02:06:57.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.380048 21638 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.14.125" port: 33787 } attempt: 1
02:06:57.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:57.381886 21638 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.21.14.125:33787: Network error: Client connection negotiation failed: client connection to 127.21.14.125:33787: connect: Connection refused (error 111)
02:06:57.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.450114 21638 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.14.125" port: 33787 } attempt: 2
02:06:57.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:57.452070 21638 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.21.14.125:33787: Network error: Client connection negotiation failed: client connection to 127.21.14.125:33787: connect: Connection refused (error 111)
02:06:57.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.543299 21638 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.14.125" port: 33787 } attempt: 3
02:06:57.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:57.545528 21638 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.21.14.125:33787: Network error: Client connection negotiation failed: client connection to 127.21.14.125:33787: connect: Connection refused (error 111)
02:06:57.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:57.612700 21640 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:06:57.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:57.612974 21640 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:06:57.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:57.613050 21640 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:06:57.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:57.621079 21640 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
02:06:57.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:57.621155 21640 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:06:57.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:57.621206 21640 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
02:06:57.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:57.621243 21640 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
02:06:57.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:57.621292 21640 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
02:06:57.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:57.621323 21640 flags.cc:432] Enabled experimental flag: --jwks_url=http://localhost:39027/jwks.json
02:06:57.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.636751 21640 master_runner.cc:387] Master server non-default flags:
02:06:57.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:06:57.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.14.84:44189
02:06:57.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:06:57.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:06:57.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/master-1/data
02:06:57.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/master-1/wal
02:06:57.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
02:06:57.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.21.14.126:34291,127.21.14.125:33787,127.21.14.124:36603
02:06:57.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:06:57.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:06:57.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
02:06:57.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.14.125:33787
02:06:57.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
02:06:57.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:06:57.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
02:06:57.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:39027/jwks.json
02:06:57.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:06:57.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:06:57.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/master-1/data/info.pb
02:06:57.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.14.125
02:06:57.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:06:57.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:06:57.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:06:57.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:06:57.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:06:57.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:06:57.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/master-1/logs
02:06:57.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:06:57.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:06:57.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:06:57.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
02:06:57.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:06:57.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
02:06:57.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:06:57.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:43:30 UTC on bdcb31816ec0
02:06:57.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12766
02:06:57.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
02:06:57.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.638768 21640 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:06:57.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.641000 21640 file_cache.cc:504] Constructed file cache file cache with capacity 419430
02:06:57.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:57.649039 21648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:06:57.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:57.649063 21649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:06:57.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:57.649919 21651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:06:57.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.650693 21640 server_base.cc:1061] running on GCE node
02:06:57.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.651508 21640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:06:57.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.652971 21640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:06:57.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.654173 21640 hybrid_clock.cc:648] HybridClock initialized: now 1783476417654122 us; error 43 us; skew 500 ppm
02:06:57.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.654561 21640 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:06:57.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.656639 21640 webserver.cc:533] Webserver started at http://127.21.14.125:35241/ using document root <none> and password file <none>
02:06:57.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.657420 21640 fs_manager.cc:362] Metadata directory not provided
02:06:57.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.657523 21640 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:06:57.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.658020 21640 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
02:06:57.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.660373 21640 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/master-1/data/instance:
02:06:57.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "3d7a486f05ad4a6aa2130e3a04ef1823"
02:06:57.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:06:57 on dist-test-slave-545r"
02:06:57.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.661326 21640 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/master-1/wal/instance:
02:06:57.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "3d7a486f05ad4a6aa2130e3a04ef1823"
02:06:57.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:06:57 on dist-test-slave-545r"
02:06:57.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.666620 21640 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.001s	sys 0.004s
02:06:57.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.670010 21657 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:06:57.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.671442 21640 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
02:06:57.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.671597 21640 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/master-1/data,/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/master-1/wal
02:06:57.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "3d7a486f05ad4a6aa2130e3a04ef1823"
02:06:57.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:06:57 on dist-test-slave-545r"
02:06:57.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.671820 21640 fs_report.cc:389] FS layout report
02:06:57.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:06:57.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/master-1/wal
02:06:57.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/master-1/wal
02:06:57.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/master-1/data/data
02:06:57.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:06:57.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:06:57.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:06:57.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:06:57.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:06:57.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:06:57.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:06:57.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:06:57.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:06:57.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:06:57.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:06:57.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.688751 21638 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.14.125" port: 33787 } attempt: 4
02:06:57.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:57.690471 21638 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.21.14.125:33787: Network error: Client connection negotiation failed: client connection to 127.21.14.125:33787: connect: Connection refused (error 111)
02:06:57.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.699585 21640 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:06:57.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.700418 21640 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:06:57.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.700664 21640 kserver.cc:163] Server-wide thread pool size limit: 3276
02:06:57.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.718333 21640 rpc_server.cc:307] RPC server started. Bound to: 127.21.14.125:33787
02:06:57.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.718351 21708 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.14.125:33787 every 8 connection(s)
02:06:57.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.719887 21640 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/master-1/data/info.pb
02:06:57.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.720595 21561 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu as pid 21640
02:06:57.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.720757 21561 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/master-1/wal/instance
02:06:57.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.721860 21561 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu
02:06:57.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu
02:06:57.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/master-2/wal
02:06:57.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/master-2/data
02:06:57.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:06:57.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:06:57.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:06:57.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:06:57.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:06:57.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:06:57.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/master-2/logs
02:06:57.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/master-2/data/info.pb
02:06:57.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:06:57.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:06:57.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:06:57.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:06:57.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:06:57.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:06:57.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:06:57.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:06:57.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
02:06:57.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:06:57.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
02:06:57.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
02:06:57.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.14.124:36603
02:06:57.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.14.124
02:06:57.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:06:57.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.14.84:44189
02:06:57.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:06:57.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:06:57.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:06:57.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
02:06:57.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.21.14.126:34291,127.21.14.125:33787,127.21.14.124:36603
02:06:57.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
02:06:57.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:39027/jwks.json with env {}
02:06:57.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.724450 21709 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
02:06:57.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.732496 21709 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.14.126" port: 34291 } has no permanent_uuid. Determining permanent_uuid...
02:06:57.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.746487 21709 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.14.125" port: 33787 } has no permanent_uuid. Determining permanent_uuid...
02:06:57.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.754702 21709 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.14.124" port: 36603 } has no permanent_uuid. Determining permanent_uuid...
02:06:57.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:57.755602 21661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.14.124:36603: connect: Connection refused (error 111)
02:06:57.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:57.757105 21709 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.21.14.124:36603: Network error: Client connection negotiation failed: client connection to 127.21.14.124:36603: connect: Connection refused (error 111)
02:06:57.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.806339 21709 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.14.124" port: 36603 } attempt: 1
02:06:57.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:57.808110 21709 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.21.14.124:36603: Network error: Client connection negotiation failed: client connection to 127.21.14.124:36603: connect: Connection refused (error 111)
02:06:57.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.876369 21709 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.14.124" port: 36603 } attempt: 2
02:06:57.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:57.878557 21709 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.21.14.124:36603: Network error: Client connection negotiation failed: client connection to 127.21.14.124:36603: connect: Connection refused (error 111)
02:06:57.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.969769 21709 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.14.124" port: 36603 } attempt: 3
02:06:57.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:57.971832 21709 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.21.14.124:36603: Network error: Client connection negotiation failed: client connection to 127.21.14.124:36603: connect: Connection refused (error 111)
02:06:57.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.989701 21638 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.14.125" port: 33787 } attempt: 5
02:06:57.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:57.995779 21638 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.14.124" port: 36603 } has no permanent_uuid. Determining permanent_uuid...
02:06:57.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:57.997516 21638 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.21.14.124:36603: Network error: Client connection negotiation failed: client connection to 127.21.14.124:36603: connect: Connection refused (error 111)
02:06:58.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:58.030666 21711 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:06:58.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:58.030921 21711 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:06:58.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:58.030999 21711 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:06:58.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:58.039032 21711 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
02:06:58.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:58.039112 21711 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:06:58.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:58.039163 21711 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
02:06:58.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:58.039201 21711 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
02:06:58.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:58.039247 21711 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
02:06:58.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:58.039278 21711 flags.cc:432] Enabled experimental flag: --jwks_url=http://localhost:39027/jwks.json
02:06:58.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.048683 21638 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.14.124" port: 36603 } attempt: 1
02:06:58.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.049551 21711 master_runner.cc:387] Master server non-default flags:
02:06:58.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:06:58.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.14.84:44189
02:06:58.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:06:58.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:06:58.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/master-2/data
02:06:58.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/master-2/wal
02:06:58.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
02:06:58.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.21.14.126:34291,127.21.14.125:33787,127.21.14.124:36603
02:06:58.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:06:58.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:06:58.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
02:06:58.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.14.124:36603
02:06:58.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
02:06:58.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:06:58.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
02:06:58.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:39027/jwks.json
02:06:58.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:06:58.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:06:58.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/master-2/data/info.pb
02:06:58.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.14.124
02:06:58.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:06:58.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:06:58.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:06:58.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:06:58.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:06:58.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:06:58.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/master-2/logs
02:06:58.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:06:58.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:06:58.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:06:58.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
02:06:58.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:06:58.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
02:06:58.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:06:58.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:43:30 UTC on bdcb31816ec0
02:06:58.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12766
02:06:58.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
02:06:58.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:58.050716 21638 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.21.14.124:36603: Network error: Client connection negotiation failed: client connection to 127.21.14.124:36603: connect: Connection refused (error 111)
02:06:58.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.051275 21711 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:06:58.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.052942 21711 file_cache.cc:504] Constructed file cache file cache with capacity 419430
02:06:58.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:58.060909 21719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:06:58.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:58.062175 21720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:06:58.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:58.063314 21722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:06:58.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.063964 21711 server_base.cc:1061] running on GCE node
02:06:58.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.064735 21711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:06:58.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.065935 21711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:06:58.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.067133 21711 hybrid_clock.cc:648] HybridClock initialized: now 1783476418067084 us; error 35 us; skew 500 ppm
02:06:58.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.067517 21711 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:06:58.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.069577 21711 webserver.cc:533] Webserver started at http://127.21.14.124:46349/ using document root <none> and password file <none>
02:06:58.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.070293 21711 fs_manager.cc:362] Metadata directory not provided
02:06:58.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.070390 21711 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:06:58.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.070803 21711 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
02:06:58.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.073082 21711 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/master-2/data/instance:
02:06:58.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "abd3346049364eccaaef66106c30e068"
02:06:58.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:06:58 on dist-test-slave-545r"
02:06:58.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.073942 21711 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/master-2/wal/instance:
02:06:58.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "abd3346049364eccaaef66106c30e068"
02:06:58.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:06:58 on dist-test-slave-545r"
02:06:58.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.078980 21711 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
02:06:58.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.082106 21728 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:06:58.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.083367 21711 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
02:06:58.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.083499 21711 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/master-2/data,/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/master-2/wal
02:06:58.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "abd3346049364eccaaef66106c30e068"
02:06:58.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:06:58 on dist-test-slave-545r"
02:06:58.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.083654 21711 fs_report.cc:389] FS layout report
02:06:58.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:06:58.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/master-2/wal
02:06:58.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/master-2/wal
02:06:58.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/master-2/data/data
02:06:58.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:06:58.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:06:58.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:06:58.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:06:58.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:06:58.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:06:58.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:06:58.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:06:58.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:06:58.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:06:58.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:06:58.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.096014 21711 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:06:58.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.096782 21711 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:06:58.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.097050 21711 kserver.cc:163] Server-wide thread pool size limit: 3276
02:06:58.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.114266 21711 rpc_server.cc:307] RPC server started. Bound to: 127.21.14.124:36603
02:06:58.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.114285 21779 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.14.124:36603 every 8 connection(s)
02:06:58.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.115031 21709 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.14.124" port: 36603 } attempt: 4
02:06:58.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.115797 21711 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/master-2/data/info.pb
02:06:58.117 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.117311 21561 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu as pid 21711
02:06:58.117 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.117460 21561 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/master-2/wal/instance
02:06:58.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.118916 21638 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.14.124" port: 36603 } attempt: 2
02:06:58.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.122025 21780 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
02:06:58.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.130259 21780 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.14.126" port: 34291 } has no permanent_uuid. Determining permanent_uuid...
02:06:58.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.137965 21709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d7a486f05ad4a6aa2130e3a04ef1823: Bootstrap starting.
02:06:58.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.142154 21709 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d7a486f05ad4a6aa2130e3a04ef1823: Neither blocks nor log segments found. Creating new log.
02:06:58.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.143147 21638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 492faeffff9b4345a5b964c4d6f2b7e6: Bootstrap starting.
02:06:58.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.143761 21709 log.cc:826] T 00000000000000000000000000000000 P 3d7a486f05ad4a6aa2130e3a04ef1823: Log is configured to *not* fsync() on all Append() calls
02:06:58.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.146471 21780 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.14.125" port: 33787 } has no permanent_uuid. Determining permanent_uuid...
02:06:58.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.148191 21709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d7a486f05ad4a6aa2130e3a04ef1823: No bootstrap required, opened a new log
02:06:58.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.148833 21638 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 492faeffff9b4345a5b964c4d6f2b7e6: Neither blocks nor log segments found. Creating new log.
02:06:58.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.150589 21638 log.cc:826] T 00000000000000000000000000000000 P 492faeffff9b4345a5b964c4d6f2b7e6: Log is configured to *not* fsync() on all Append() calls
02:06:58.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.157064 21780 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.14.124" port: 36603 } has no permanent_uuid. Determining permanent_uuid...
02:06:58.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.158342 21709 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3d7a486f05ad4a6aa2130e3a04ef1823 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492faeffff9b4345a5b964c4d6f2b7e6" member_type: VOTER last_known_addr { host: "127.21.14.126" port: 34291 } } peers { permanent_uuid: "3d7a486f05ad4a6aa2130e3a04ef1823" member_type: VOTER last_known_addr { host: "127.21.14.125" port: 33787 } } peers { permanent_uuid: "abd3346049364eccaaef66106c30e068" member_type: VOTER last_known_addr { host: "127.21.14.124" port: 36603 } }
02:06:58.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.158983 21709 raft_consensus.cc:385] T 00000000000000000000000000000000 P 3d7a486f05ad4a6aa2130e3a04ef1823 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
02:06:58.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.159368 21709 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3d7a486f05ad4a6aa2130e3a04ef1823 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d7a486f05ad4a6aa2130e3a04ef1823, State: Initialized, Role: FOLLOWER
02:06:58.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.160391 21709 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d7a486f05ad4a6aa2130e3a04ef1823 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492faeffff9b4345a5b964c4d6f2b7e6" member_type: VOTER last_known_addr { host: "127.21.14.126" port: 34291 } } peers { permanent_uuid: "3d7a486f05ad4a6aa2130e3a04ef1823" member_type: VOTER last_known_addr { host: "127.21.14.125" port: 33787 } } peers { permanent_uuid: "abd3346049364eccaaef66106c30e068" member_type: VOTER last_known_addr { host: "127.21.14.124" port: 36603 } }
02:06:58.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.161203 21638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 492faeffff9b4345a5b964c4d6f2b7e6: No bootstrap required, opened a new log
02:06:58.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.165299 21709 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3d7a486f05ad4a6aa2130e3a04ef1823 [sys.catalog]: configured and running, proceeding with master startup.
02:06:58.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.165135 21790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d7a486f05ad4a6aa2130e3a04ef1823 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492faeffff9b4345a5b964c4d6f2b7e6" member_type: VOTER last_known_addr { host: "127.21.14.126" port: 34291 } } peers { permanent_uuid: "3d7a486f05ad4a6aa2130e3a04ef1823" member_type: VOTER last_known_addr { host: "127.21.14.125" port: 33787 } } peers { permanent_uuid: "abd3346049364eccaaef66106c30e068" member_type: VOTER last_known_addr { host: "127.21.14.124" port: 36603 } } }
02:06:58.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.166033 21790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d7a486f05ad4a6aa2130e3a04ef1823 [sys.catalog]: This master's current role is: FOLLOWER
02:06:58.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.168123 21638 raft_consensus.cc:359] T 00000000000000000000000000000000 P 492faeffff9b4345a5b964c4d6f2b7e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492faeffff9b4345a5b964c4d6f2b7e6" member_type: VOTER last_known_addr { host: "127.21.14.126" port: 34291 } } peers { permanent_uuid: "3d7a486f05ad4a6aa2130e3a04ef1823" member_type: VOTER last_known_addr { host: "127.21.14.125" port: 33787 } } peers { permanent_uuid: "abd3346049364eccaaef66106c30e068" member_type: VOTER last_known_addr { host: "127.21.14.124" port: 36603 } }
02:06:58.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.168409 21638 raft_consensus.cc:385] T 00000000000000000000000000000000 P 492faeffff9b4345a5b964c4d6f2b7e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
02:06:58.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.168840 21638 raft_consensus.cc:740] T 00000000000000000000000000000000 P 492faeffff9b4345a5b964c4d6f2b7e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492faeffff9b4345a5b964c4d6f2b7e6, State: Initialized, Role: FOLLOWER
02:06:58.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.169821 21638 consensus_queue.cc:260] T 00000000000000000000000000000000 P 492faeffff9b4345a5b964c4d6f2b7e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492faeffff9b4345a5b964c4d6f2b7e6" member_type: VOTER last_known_addr { host: "127.21.14.126" port: 34291 } } peers { permanent_uuid: "3d7a486f05ad4a6aa2130e3a04ef1823" member_type: VOTER last_known_addr { host: "127.21.14.125" port: 33787 } } peers { permanent_uuid: "abd3346049364eccaaef66106c30e068" member_type: VOTER last_known_addr { host: "127.21.14.124" port: 36603 } }
02:06:58.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.171198 21794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 492faeffff9b4345a5b964c4d6f2b7e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492faeffff9b4345a5b964c4d6f2b7e6" member_type: VOTER last_known_addr { host: "127.21.14.126" port: 34291 } } peers { permanent_uuid: "3d7a486f05ad4a6aa2130e3a04ef1823" member_type: VOTER last_known_addr { host: "127.21.14.125" port: 33787 } } peers { permanent_uuid: "abd3346049364eccaaef66106c30e068" member_type: VOTER last_known_addr { host: "127.21.14.124" port: 36603 } } }
02:06:58.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.171813 21794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 492faeffff9b4345a5b964c4d6f2b7e6 [sys.catalog]: This master's current role is: FOLLOWER
02:06:58.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.172294 21638 sys_catalog.cc:565] T 00000000000000000000000000000000 P 492faeffff9b4345a5b964c4d6f2b7e6 [sys.catalog]: configured and running, proceeding with master startup.
02:06:58.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.173177 21780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abd3346049364eccaaef66106c30e068: Bootstrap starting.
02:06:58.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.178233 21780 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abd3346049364eccaaef66106c30e068: Neither blocks nor log segments found. Creating new log.
02:06:58.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.179878 21780 log.cc:826] T 00000000000000000000000000000000 P abd3346049364eccaaef66106c30e068: Log is configured to *not* fsync() on all Append() calls
02:06:58.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.182152 21640 master_runner.cc:187] Error getting master registration for 127.21.14.126:34291: OK, instance_id { permanent_uuid: "492faeffff9b4345a5b964c4d6f2b7e6" instance_seqno: 1783476417295391 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
02:06:58.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.188942 21640 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
02:06:58.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.195168 21780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abd3346049364eccaaef66106c30e068: No bootstrap required, opened a new log
02:06:58.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:58.202659 21663 tablet.cc:2406] T 00000000000000000000000000000000 P 3d7a486f05ad4a6aa2130e3a04ef1823: Can't schedule compaction. Clean time has not been advanced past its initial value.
02:06:58.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.204375 21780 raft_consensus.cc:359] T 00000000000000000000000000000000 P abd3346049364eccaaef66106c30e068 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492faeffff9b4345a5b964c4d6f2b7e6" member_type: VOTER last_known_addr { host: "127.21.14.126" port: 34291 } } peers { permanent_uuid: "3d7a486f05ad4a6aa2130e3a04ef1823" member_type: VOTER last_known_addr { host: "127.21.14.125" port: 33787 } } peers { permanent_uuid: "abd3346049364eccaaef66106c30e068" member_type: VOTER last_known_addr { host: "127.21.14.124" port: 36603 } }
02:06:58.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.205066 21780 raft_consensus.cc:385] T 00000000000000000000000000000000 P abd3346049364eccaaef66106c30e068 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
02:06:58.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.205437 21780 raft_consensus.cc:740] T 00000000000000000000000000000000 P abd3346049364eccaaef66106c30e068 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abd3346049364eccaaef66106c30e068, State: Initialized, Role: FOLLOWER
02:06:58.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.206334 21780 consensus_queue.cc:260] T 00000000000000000000000000000000 P abd3346049364eccaaef66106c30e068 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492faeffff9b4345a5b964c4d6f2b7e6" member_type: VOTER last_known_addr { host: "127.21.14.126" port: 34291 } } peers { permanent_uuid: "3d7a486f05ad4a6aa2130e3a04ef1823" member_type: VOTER last_known_addr { host: "127.21.14.125" port: 33787 } } peers { permanent_uuid: "abd3346049364eccaaef66106c30e068" member_type: VOTER last_known_addr { host: "127.21.14.124" port: 36603 } }
02:06:58.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.200099 21572 master_runner.cc:187] Error getting master registration for 127.21.14.124:36603: OK, instance_id { permanent_uuid: "abd3346049364eccaaef66106c30e068" instance_seqno: 1783476418109860 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
02:06:58.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.209388 21572 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
02:06:58.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:58.209887 21811 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 3d7a486f05ad4a6aa2130e3a04ef1823: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
02:06:58.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:58.210225 21811 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
02:06:58.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:58.213160 21815 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 492faeffff9b4345a5b964c4d6f2b7e6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
02:06:58.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:58.213914 21815 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
02:06:58.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.216214 21817 sys_catalog.cc:455] T 00000000000000000000000000000000 P abd3346049364eccaaef66106c30e068 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492faeffff9b4345a5b964c4d6f2b7e6" member_type: VOTER last_known_addr { host: "127.21.14.126" port: 34291 } } peers { permanent_uuid: "3d7a486f05ad4a6aa2130e3a04ef1823" member_type: VOTER last_known_addr { host: "127.21.14.125" port: 33787 } } peers { permanent_uuid: "abd3346049364eccaaef66106c30e068" member_type: VOTER last_known_addr { host: "127.21.14.124" port: 36603 } } }
02:06:58.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.216903 21817 sys_catalog.cc:458] T 00000000000000000000000000000000 P abd3346049364eccaaef66106c30e068 [sys.catalog]: This master's current role is: FOLLOWER
02:06:58.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.217370 21780 sys_catalog.cc:565] T 00000000000000000000000000000000 P abd3346049364eccaaef66106c30e068 [sys.catalog]: configured and running, proceeding with master startup.
02:06:58.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.234823 21711 master_runner.cc:264] No leader master found from master abd3346049364eccaaef66106c30e068
02:06:58.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.234930 21711 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
02:06:58.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:58.243567 21828 catalog_manager.cc:1594] T 00000000000000000000000000000000 P abd3346049364eccaaef66106c30e068: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
02:06:58.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:58.243937 21828 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
02:06:58.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:58.284768 21592 tablet.cc:2406] T 00000000000000000000000000000000 P 492faeffff9b4345a5b964c4d6f2b7e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
02:06:58.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.318003 21794 raft_consensus.cc:493] T 00000000000000000000000000000000 P 492faeffff9b4345a5b964c4d6f2b7e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
02:06:58.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.318266 21794 raft_consensus.cc:515] T 00000000000000000000000000000000 P 492faeffff9b4345a5b964c4d6f2b7e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492faeffff9b4345a5b964c4d6f2b7e6" member_type: VOTER last_known_addr { host: "127.21.14.126" port: 34291 } } peers { permanent_uuid: "3d7a486f05ad4a6aa2130e3a04ef1823" member_type: VOTER last_known_addr { host: "127.21.14.125" port: 33787 } } peers { permanent_uuid: "abd3346049364eccaaef66106c30e068" member_type: VOTER last_known_addr { host: "127.21.14.124" port: 36603 } }
02:06:58.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.319949 21794 leader_election.cc:290] T 00000000000000000000000000000000 P 492faeffff9b4345a5b964c4d6f2b7e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d7a486f05ad4a6aa2130e3a04ef1823 (127.21.14.125:33787), abd3346049364eccaaef66106c30e068 (127.21.14.124:36603)
02:06:58.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.322204 21684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "492faeffff9b4345a5b964c4d6f2b7e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d7a486f05ad4a6aa2130e3a04ef1823" is_pre_election: true
02:06:58.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.322674 21684 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 3d7a486f05ad4a6aa2130e3a04ef1823 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492faeffff9b4345a5b964c4d6f2b7e6 in term 0.
02:06:58.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.323597 21755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "492faeffff9b4345a5b964c4d6f2b7e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abd3346049364eccaaef66106c30e068" is_pre_election: true
02:06:58.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.324072 21755 raft_consensus.cc:2468] T 00000000000000000000000000000000 P abd3346049364eccaaef66106c30e068 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492faeffff9b4345a5b964c4d6f2b7e6 in term 0.
02:06:58.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.326416 21590 leader_election.cc:304] T 00000000000000000000000000000000 P 492faeffff9b4345a5b964c4d6f2b7e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492faeffff9b4345a5b964c4d6f2b7e6, abd3346049364eccaaef66106c30e068; no voters: 
02:06:58.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.329011 21794 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 492faeffff9b4345a5b964c4d6f2b7e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
02:06:58.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.329144 21794 raft_consensus.cc:493] T 00000000000000000000000000000000 P 492faeffff9b4345a5b964c4d6f2b7e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
02:06:58.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.329228 21794 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 492faeffff9b4345a5b964c4d6f2b7e6 [term 0 FOLLOWER]: Advancing to term 1
02:06:58.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.332440 21794 raft_consensus.cc:515] T 00000000000000000000000000000000 P 492faeffff9b4345a5b964c4d6f2b7e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492faeffff9b4345a5b964c4d6f2b7e6" member_type: VOTER last_known_addr { host: "127.21.14.126" port: 34291 } } peers { permanent_uuid: "3d7a486f05ad4a6aa2130e3a04ef1823" member_type: VOTER last_known_addr { host: "127.21.14.125" port: 33787 } } peers { permanent_uuid: "abd3346049364eccaaef66106c30e068" member_type: VOTER last_known_addr { host: "127.21.14.124" port: 36603 } }
02:06:58.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.335180 21684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "492faeffff9b4345a5b964c4d6f2b7e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d7a486f05ad4a6aa2130e3a04ef1823"
02:06:58.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.335366 21684 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3d7a486f05ad4a6aa2130e3a04ef1823 [term 0 FOLLOWER]: Advancing to term 1
02:06:58.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.338709 21684 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 3d7a486f05ad4a6aa2130e3a04ef1823 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492faeffff9b4345a5b964c4d6f2b7e6 in term 1.
02:06:58.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.339084 21589 leader_election.cc:304] T 00000000000000000000000000000000 P 492faeffff9b4345a5b964c4d6f2b7e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d7a486f05ad4a6aa2130e3a04ef1823, 492faeffff9b4345a5b964c4d6f2b7e6; no voters: 
02:06:58.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.342950 21755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "492faeffff9b4345a5b964c4d6f2b7e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abd3346049364eccaaef66106c30e068"
02:06:58.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.343142 21755 raft_consensus.cc:3060] T 00000000000000000000000000000000 P abd3346049364eccaaef66106c30e068 [term 0 FOLLOWER]: Advancing to term 1
02:06:58.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.346560 21755 raft_consensus.cc:2468] T 00000000000000000000000000000000 P abd3346049364eccaaef66106c30e068 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492faeffff9b4345a5b964c4d6f2b7e6 in term 1.
02:06:58.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:58.349097 21734 tablet.cc:2406] T 00000000000000000000000000000000 P abd3346049364eccaaef66106c30e068: Can't schedule compaction. Clean time has not been advanced past its initial value.
02:06:58.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.350726 21794 leader_election.cc:290] T 00000000000000000000000000000000 P 492faeffff9b4345a5b964c4d6f2b7e6 [CANDIDATE]: Term 1 election: Requested vote from peers 3d7a486f05ad4a6aa2130e3a04ef1823 (127.21.14.125:33787), abd3346049364eccaaef66106c30e068 (127.21.14.124:36603)
02:06:58.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.350939 21794 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 492faeffff9b4345a5b964c4d6f2b7e6 [term 1 FOLLOWER]: Leader election won for term 1
02:06:58.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.351055 21794 raft_consensus.cc:697] T 00000000000000000000000000000000 P 492faeffff9b4345a5b964c4d6f2b7e6 [term 1 LEADER]: Becoming Leader. State: Replica: 492faeffff9b4345a5b964c4d6f2b7e6, State: Running, Role: LEADER
02:06:58.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.351529 21794 consensus_queue.cc:237] T 00000000000000000000000000000000 P 492faeffff9b4345a5b964c4d6f2b7e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492faeffff9b4345a5b964c4d6f2b7e6" member_type: VOTER last_known_addr { host: "127.21.14.126" port: 34291 } } peers { permanent_uuid: "3d7a486f05ad4a6aa2130e3a04ef1823" member_type: VOTER last_known_addr { host: "127.21.14.125" port: 33787 } } peers { permanent_uuid: "abd3346049364eccaaef66106c30e068" member_type: VOTER last_known_addr { host: "127.21.14.124" port: 36603 } }
02:06:58.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.371505 21830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 492faeffff9b4345a5b964c4d6f2b7e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 492faeffff9b4345a5b964c4d6f2b7e6. Latest consensus state: current_term: 1 leader_uuid: "492faeffff9b4345a5b964c4d6f2b7e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492faeffff9b4345a5b964c4d6f2b7e6" member_type: VOTER last_known_addr { host: "127.21.14.126" port: 34291 } } peers { permanent_uuid: "3d7a486f05ad4a6aa2130e3a04ef1823" member_type: VOTER last_known_addr { host: "127.21.14.125" port: 33787 } } peers { permanent_uuid: "abd3346049364eccaaef66106c30e068" member_type: VOTER last_known_addr { host: "127.21.14.124" port: 36603 } } }
02:06:58.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.372352 21830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 492faeffff9b4345a5b964c4d6f2b7e6 [sys.catalog]: This master's current role is: LEADER
02:06:58.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.379763 21832 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
02:06:58.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.382404 21832 catalog_manager.cc:1520] Initializing Kudu cluster ID...
02:06:58.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.400925 21755 raft_consensus.cc:1275] T 00000000000000000000000000000000 P abd3346049364eccaaef66106c30e068 [term 1 FOLLOWER]: Refusing update from remote peer 492faeffff9b4345a5b964c4d6f2b7e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
02:06:58.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.402215 21830 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 492faeffff9b4345a5b964c4d6f2b7e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abd3346049364eccaaef66106c30e068" member_type: VOTER last_known_addr { host: "127.21.14.124" port: 36603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
02:06:58.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.409482 21684 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 3d7a486f05ad4a6aa2130e3a04ef1823 [term 1 FOLLOWER]: Refusing update from remote peer 492faeffff9b4345a5b964c4d6f2b7e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
02:06:58.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.413223 21830 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 492faeffff9b4345a5b964c4d6f2b7e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d7a486f05ad4a6aa2130e3a04ef1823" member_type: VOTER last_known_addr { host: "127.21.14.125" port: 33787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
02:06:58.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.446914 21790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d7a486f05ad4a6aa2130e3a04ef1823 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 492faeffff9b4345a5b964c4d6f2b7e6. Latest consensus state: current_term: 1 leader_uuid: "492faeffff9b4345a5b964c4d6f2b7e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492faeffff9b4345a5b964c4d6f2b7e6" member_type: VOTER last_known_addr { host: "127.21.14.126" port: 34291 } } peers { permanent_uuid: "3d7a486f05ad4a6aa2130e3a04ef1823" member_type: VOTER last_known_addr { host: "127.21.14.125" port: 33787 } } peers { permanent_uuid: "abd3346049364eccaaef66106c30e068" member_type: VOTER last_known_addr { host: "127.21.14.124" port: 36603 } } }
02:06:58.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.447184 21790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d7a486f05ad4a6aa2130e3a04ef1823 [sys.catalog]: This master's current role is: FOLLOWER
02:06:58.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.450676 21833 mvcc.cc:204] Tried to move back new op lower bound from 7305119409758924800 to 7305119409584009216. Current Snapshot: MvccSnapshot[applied={T|T < 7305119409758924800}]
02:06:58.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.455642 21835 mvcc.cc:204] Tried to move back new op lower bound from 7305119409758924800 to 7305119409584009216. Current Snapshot: MvccSnapshot[applied={T|T < 7305119409758924800}]
02:06:58.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.460882 21794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 492faeffff9b4345a5b964c4d6f2b7e6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "492faeffff9b4345a5b964c4d6f2b7e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492faeffff9b4345a5b964c4d6f2b7e6" member_type: VOTER last_known_addr { host: "127.21.14.126" port: 34291 } } peers { permanent_uuid: "3d7a486f05ad4a6aa2130e3a04ef1823" member_type: VOTER last_known_addr { host: "127.21.14.125" port: 33787 } } peers { permanent_uuid: "abd3346049364eccaaef66106c30e068" member_type: VOTER last_known_addr { host: "127.21.14.124" port: 36603 } } }
02:06:58.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.461167 21794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 492faeffff9b4345a5b964c4d6f2b7e6 [sys.catalog]: This master's current role is: LEADER
02:06:58.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.463006 21832 catalog_manager.cc:1383] Generated new cluster ID: 11f353e998ba4aaa8d47caf3d92e3727
02:06:58.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.463099 21832 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
02:06:58.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.466833 21817 sys_catalog.cc:455] T 00000000000000000000000000000000 P abd3346049364eccaaef66106c30e068 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 492faeffff9b4345a5b964c4d6f2b7e6. Latest consensus state: current_term: 1 leader_uuid: "492faeffff9b4345a5b964c4d6f2b7e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492faeffff9b4345a5b964c4d6f2b7e6" member_type: VOTER last_known_addr { host: "127.21.14.126" port: 34291 } } peers { permanent_uuid: "3d7a486f05ad4a6aa2130e3a04ef1823" member_type: VOTER last_known_addr { host: "127.21.14.125" port: 33787 } } peers { permanent_uuid: "abd3346049364eccaaef66106c30e068" member_type: VOTER last_known_addr { host: "127.21.14.124" port: 36603 } } }
02:06:58.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.467067 21817 sys_catalog.cc:458] T 00000000000000000000000000000000 P abd3346049364eccaaef66106c30e068 [sys.catalog]: This master's current role is: FOLLOWER
02:06:58.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.471813 21790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d7a486f05ad4a6aa2130e3a04ef1823 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "492faeffff9b4345a5b964c4d6f2b7e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492faeffff9b4345a5b964c4d6f2b7e6" member_type: VOTER last_known_addr { host: "127.21.14.126" port: 34291 } } peers { permanent_uuid: "3d7a486f05ad4a6aa2130e3a04ef1823" member_type: VOTER last_known_addr { host: "127.21.14.125" port: 33787 } } peers { permanent_uuid: "abd3346049364eccaaef66106c30e068" member_type: VOTER last_known_addr { host: "127.21.14.124" port: 36603 } } }
02:06:58.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.472014 21790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d7a486f05ad4a6aa2130e3a04ef1823 [sys.catalog]: This master's current role is: FOLLOWER
02:06:58.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.472162 21830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 492faeffff9b4345a5b964c4d6f2b7e6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "492faeffff9b4345a5b964c4d6f2b7e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492faeffff9b4345a5b964c4d6f2b7e6" member_type: VOTER last_known_addr { host: "127.21.14.126" port: 34291 } } peers { permanent_uuid: "3d7a486f05ad4a6aa2130e3a04ef1823" member_type: VOTER last_known_addr { host: "127.21.14.125" port: 33787 } } peers { permanent_uuid: "abd3346049364eccaaef66106c30e068" member_type: VOTER last_known_addr { host: "127.21.14.124" port: 36603 } } }
02:06:58.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.472353 21830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 492faeffff9b4345a5b964c4d6f2b7e6 [sys.catalog]: This master's current role is: LEADER
02:06:58.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.484274 21817 sys_catalog.cc:455] T 00000000000000000000000000000000 P abd3346049364eccaaef66106c30e068 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "492faeffff9b4345a5b964c4d6f2b7e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492faeffff9b4345a5b964c4d6f2b7e6" member_type: VOTER last_known_addr { host: "127.21.14.126" port: 34291 } } peers { permanent_uuid: "3d7a486f05ad4a6aa2130e3a04ef1823" member_type: VOTER last_known_addr { host: "127.21.14.125" port: 33787 } } peers { permanent_uuid: "abd3346049364eccaaef66106c30e068" member_type: VOTER last_known_addr { host: "127.21.14.124" port: 36603 } } }
02:06:58.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.484452 21817 sys_catalog.cc:458] T 00000000000000000000000000000000 P abd3346049364eccaaef66106c30e068 [sys.catalog]: This master's current role is: FOLLOWER
02:06:58.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.491240 21834 mvcc.cc:204] Tried to move back new op lower bound from 7305119409758924800 to 7305119409584009216. Current Snapshot: MvccSnapshot[applied={T|T < 7305119409758924800 or (T in {7305119409758924800})}]
02:06:58.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.495254 21832 catalog_manager.cc:1406] Generated new certificate authority record
02:06:58.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.496939 21832 catalog_manager.cc:1540] Loading token signing keys...
02:06:58.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.537147 21832 catalog_manager.cc:6070] T 00000000000000000000000000000000 P 492faeffff9b4345a5b964c4d6f2b7e6: Generated new TSK 0
02:06:58.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.539981 21832 catalog_manager.cc:1550] Initializing in-progress tserver states...
02:06:58.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.566905 21561 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu
02:06:58.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu
02:06:58.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-0/wal
02:06:58.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-0/data
02:06:58.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:06:58.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:06:58.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:06:58.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:06:58.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:06:58.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:06:58.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-0/logs
02:06:58.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-0/data/info.pb
02:06:58.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:06:58.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:06:58.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:06:58.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:06:58.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:06:58.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:06:58.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:06:58.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:06:58.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:06:58.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:06:58.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.14.65:0
02:06:58.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.21.14.65
02:06:58.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.14.65
02:06:58.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:06:58.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.21.14.126:34291,127.21.14.125:33787,127.21.14.124:36603
02:06:58.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.14.84:44189
02:06:58.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:06:58.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:06:58.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:06:58.964 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:58.963806 21842 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:06:58.964 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:58.964073 21842 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:06:58.964 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:58.964147 21842 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:06:58.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:58.972146 21842 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:06:58.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:58.972298 21842 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.14.65
02:06:58.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.982077 21842 tablet_server_runner.cc:78] Tablet server non-default flags:
02:06:58.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:06:58.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.14.84:44189
02:06:58.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:06:58.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:06:58.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-0/data
02:06:58.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-0/wal
02:06:58.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:06:58.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:06:58.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.14.65:0
02:06:58.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:06:58.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:06:58.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:06:58.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-0/data/info.pb
02:06:58.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.14.65
02:06:58.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:06:58.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.21.14.126:34291,127.21.14.125:33787,127.21.14.124:36603
02:06:58.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:06:58.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:06:58.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:06:58.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:06:58.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:06:58.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.21.14.65
02:06:58.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-0/logs
02:06:58.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:06:58.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:06:58.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:06:58.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:06:58.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:06:58.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
02:06:58.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:06:58.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:43:30 UTC on bdcb31816ec0
02:06:58.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12766
02:06:58.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
02:06:58.984 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.983991 21842 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:06:58.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:58.985982 21842 file_cache.cc:504] Constructed file cache file cache with capacity 419430
02:06:58.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:58.994592 21847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:06:58.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:58.994865 21848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:06:58.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:58.996930 21850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:06:59.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.000349 21842 server_base.cc:1061] running on GCE node
02:06:59.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.001021 21842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:06:59.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.002111 21842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:06:59.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.003309 21842 hybrid_clock.cc:648] HybridClock initialized: now 1783476419003203 us; error 97 us; skew 500 ppm
02:06:59.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.003643 21842 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:06:59.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.005718 21842 webserver.cc:533] Webserver started at http://127.21.14.65:36395/ using document root <none> and password file <none>
02:06:59.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.006428 21842 fs_manager.cc:362] Metadata directory not provided
02:06:59.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.006524 21842 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:06:59.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.006965 21842 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
02:06:59.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.009258 21842 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-0/data/instance:
02:06:59.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "790137564e884f989bbd5c36475181af"
02:06:59.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:06:59 on dist-test-slave-545r"
02:06:59.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.010116 21842 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-0/wal/instance:
02:06:59.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "790137564e884f989bbd5c36475181af"
02:06:59.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:06:59 on dist-test-slave-545r"
02:06:59.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.015100 21842 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
02:06:59.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.018296 21856 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:06:59.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.019496 21842 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
02:06:59.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.019654 21842 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-0/data,/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-0/wal
02:06:59.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "790137564e884f989bbd5c36475181af"
02:06:59.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:06:59 on dist-test-slave-545r"
02:06:59.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.019860 21842 fs_report.cc:389] FS layout report
02:06:59.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:06:59.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-0/wal
02:06:59.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-0/wal
02:06:59.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-0/data/data
02:06:59.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:06:59.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:06:59.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:06:59.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:06:59.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:06:59.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:06:59.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:06:59.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:06:59.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:06:59.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:06:59.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:06:59.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.033960 21842 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:06:59.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.034736 21842 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:06:59.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.035012 21842 kserver.cc:163] Server-wide thread pool size limit: 3276
02:06:59.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.035924 21842 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:06:59.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.037631 21842 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
02:06:59.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.037729 21842 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
02:06:59.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.037820 21842 ts_tablet_manager.cc:616] Registered 0 tablets
02:06:59.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.037897 21842 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
02:06:59.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.067227 21842 rpc_server.cc:307] RPC server started. Bound to: 127.21.14.65:38711
02:06:59.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.067283 21968 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.14.65:38711 every 8 connection(s)
02:06:59.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.068804 21842 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-0/data/info.pb
02:06:59.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.077102 21561 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu as pid 21842
02:06:59.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.077275 21561 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-0/wal/instance
02:06:59.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.087639 21561 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu
02:06:59.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu
02:06:59.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-1/wal
02:06:59.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-1/data
02:06:59.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:06:59.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:06:59.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:06:59.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:06:59.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:06:59.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:06:59.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-1/logs
02:06:59.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-1/data/info.pb
02:06:59.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:06:59.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:06:59.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:06:59.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:06:59.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:06:59.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:06:59.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:06:59.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:06:59.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:06:59.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:06:59.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.14.66:0
02:06:59.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.21.14.66
02:06:59.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.14.66
02:06:59.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:06:59.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.21.14.126:34291,127.21.14.125:33787,127.21.14.124:36603
02:06:59.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.14.84:44189
02:06:59.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:06:59.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:06:59.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:06:59.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.094090 21971 heartbeater.cc:344] Connected to a master server at 127.21.14.125:33787
02:06:59.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.094508 21971 heartbeater.cc:461] Registering TS with master...
02:06:59.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.095841 21971 heartbeater.cc:507] Master 127.21.14.125:33787 requested a full tablet report, sending...
02:06:59.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.098754 21674 ts_manager.cc:194] Registered new tserver with Master: 790137564e884f989bbd5c36475181af (127.21.14.65:38711)
02:06:59.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.099494 21969 heartbeater.cc:344] Connected to a master server at 127.21.14.124:36603
02:06:59.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.099640 21969 heartbeater.cc:461] Registering TS with master...
02:06:59.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.099920 21970 heartbeater.cc:344] Connected to a master server at 127.21.14.126:34291
02:06:59.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.100060 21970 heartbeater.cc:461] Registering TS with master...
02:06:59.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.100205 21969 heartbeater.cc:507] Master 127.21.14.124:36603 requested a full tablet report, sending...
02:06:59.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.100641 21970 heartbeater.cc:507] Master 127.21.14.126:34291 requested a full tablet report, sending...
02:06:59.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.102118 21745 ts_manager.cc:194] Registered new tserver with Master: 790137564e884f989bbd5c36475181af (127.21.14.65:38711)
02:06:59.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.102384 21603 ts_manager.cc:194] Registered new tserver with Master: 790137564e884f989bbd5c36475181af (127.21.14.65:38711)
02:06:59.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.104485 21603 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.21.14.65:38215
02:06:59.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.212141 21811 catalog_manager.cc:1295] Loaded cluster ID: 11f353e998ba4aaa8d47caf3d92e3727
02:06:59.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.212229 21811 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 3d7a486f05ad4a6aa2130e3a04ef1823: loading cluster ID for follower catalog manager: success
02:06:59.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.214721 21811 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 3d7a486f05ad4a6aa2130e3a04ef1823: acquiring CA information for follower catalog manager: success
02:06:59.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.216295 21811 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 3d7a486f05ad4a6aa2130e3a04ef1823: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
02:06:59.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.246006 21828 catalog_manager.cc:1295] Loaded cluster ID: 11f353e998ba4aaa8d47caf3d92e3727
02:06:59.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.246110 21828 catalog_manager.cc:1588] T 00000000000000000000000000000000 P abd3346049364eccaaef66106c30e068: loading cluster ID for follower catalog manager: success
02:06:59.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.248698 21828 catalog_manager.cc:1610] T 00000000000000000000000000000000 P abd3346049364eccaaef66106c30e068: acquiring CA information for follower catalog manager: success
02:06:59.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.250303 21828 catalog_manager.cc:1638] T 00000000000000000000000000000000 P abd3346049364eccaaef66106c30e068: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
02:06:59.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:59.401167 21979 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:06:59.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:59.401445 21979 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:06:59.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:59.401546 21979 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:06:59.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:59.409776 21979 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:06:59.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:59.409948 21979 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.14.66
02:06:59.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.419461 21979 tablet_server_runner.cc:78] Tablet server non-default flags:
02:06:59.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:06:59.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.14.84:44189
02:06:59.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:06:59.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:06:59.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-1/data
02:06:59.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-1/wal
02:06:59.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:06:59.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:06:59.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.14.66:0
02:06:59.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:06:59.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:06:59.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:06:59.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-1/data/info.pb
02:06:59.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.14.66
02:06:59.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:06:59.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.21.14.126:34291,127.21.14.125:33787,127.21.14.124:36603
02:06:59.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:06:59.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:06:59.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:06:59.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:06:59.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:06:59.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.21.14.66
02:06:59.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-1/logs
02:06:59.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:06:59.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:06:59.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:06:59.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:06:59.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:06:59.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
02:06:59.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:06:59.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:43:30 UTC on bdcb31816ec0
02:06:59.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12766
02:06:59.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
02:06:59.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.421543 21979 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:06:59.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.423300 21979 file_cache.cc:504] Constructed file cache file cache with capacity 419430
02:06:59.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:59.432601 21987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:06:59.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:59.432574 21984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:06:59.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:59.432799 21985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:06:59.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.433209 21979 server_base.cc:1061] running on GCE node
02:06:59.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.433946 21979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:06:59.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.434839 21979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:06:59.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.436035 21979 hybrid_clock.cc:648] HybridClock initialized: now 1783476419435985 us; error 45 us; skew 500 ppm
02:06:59.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.436360 21979 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:06:59.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.438345 21979 webserver.cc:533] Webserver started at http://127.21.14.66:43569/ using document root <none> and password file <none>
02:06:59.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.439116 21979 fs_manager.cc:362] Metadata directory not provided
02:06:59.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.439213 21979 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:06:59.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.439623 21979 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
02:06:59.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.441848 21979 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-1/data/instance:
02:06:59.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "34e2861a29784992b9d1a9a5876087a6"
02:06:59.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:06:59 on dist-test-slave-545r"
02:06:59.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.442643 21979 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-1/wal/instance:
02:06:59.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "34e2861a29784992b9d1a9a5876087a6"
02:06:59.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:06:59 on dist-test-slave-545r"
02:06:59.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.447414 21979 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.001s
02:06:59.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.450410 21993 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:06:59.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.451545 21979 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
02:06:59.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.451682 21979 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-1/data,/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-1/wal
02:06:59.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "34e2861a29784992b9d1a9a5876087a6"
02:06:59.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:06:59 on dist-test-slave-545r"
02:06:59.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.451886 21979 fs_report.cc:389] FS layout report
02:06:59.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:06:59.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-1/wal
02:06:59.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-1/wal
02:06:59.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-1/data/data
02:06:59.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:06:59.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:06:59.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:06:59.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:06:59.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:06:59.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:06:59.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:06:59.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:06:59.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:06:59.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:06:59.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:06:59.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.466852 21979 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:06:59.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.467810 21979 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:06:59.468 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.468093 21979 kserver.cc:163] Server-wide thread pool size limit: 3276
02:06:59.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.469056 21979 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:06:59.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.470693 21979 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
02:06:59.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.470785 21979 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
02:06:59.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.470902 21979 ts_tablet_manager.cc:616] Registered 0 tablets
02:06:59.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.470978 21979 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
02:06:59.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.500684 21979 rpc_server.cc:307] RPC server started. Bound to: 127.21.14.66:35979
02:06:59.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.500751 22105 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.14.66:35979 every 8 connection(s)
02:06:59.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.502190 21979 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-1/data/info.pb
02:06:59.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.505326 21561 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu as pid 21979
02:06:59.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.505504 21561 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-1/wal/instance
02:06:59.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.510092 21561 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu
02:06:59.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu
02:06:59.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/wal
02:06:59.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/data
02:06:59.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:06:59.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:06:59.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:06:59.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:06:59.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:06:59.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:06:59.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/logs
02:06:59.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/data/info.pb
02:06:59.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:06:59.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:06:59.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:06:59.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:06:59.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:06:59.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:06:59.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:06:59.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:06:59.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:06:59.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:06:59.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.14.67:0
02:06:59.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.21.14.67
02:06:59.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.14.67
02:06:59.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:06:59.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.21.14.126:34291,127.21.14.125:33787,127.21.14.124:36603
02:06:59.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.14.84:44189
02:06:59.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:06:59.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:06:59.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:06:59.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.519856 22106 heartbeater.cc:344] Connected to a master server at 127.21.14.124:36603
02:06:59.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.519917 22108 heartbeater.cc:344] Connected to a master server at 127.21.14.125:33787
02:06:59.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.520154 22106 heartbeater.cc:461] Registering TS with master...
02:06:59.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.520238 22108 heartbeater.cc:461] Registering TS with master...
02:06:59.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.521328 22106 heartbeater.cc:507] Master 127.21.14.124:36603 requested a full tablet report, sending...
02:06:59.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.521353 22108 heartbeater.cc:507] Master 127.21.14.125:33787 requested a full tablet report, sending...
02:06:59.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.521611 22107 heartbeater.cc:344] Connected to a master server at 127.21.14.126:34291
02:06:59.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.521736 22107 heartbeater.cc:461] Registering TS with master...
02:06:59.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.522241 22107 heartbeater.cc:507] Master 127.21.14.126:34291 requested a full tablet report, sending...
02:06:59.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.523002 21674 ts_manager.cc:194] Registered new tserver with Master: 34e2861a29784992b9d1a9a5876087a6 (127.21.14.66:35979)
02:06:59.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.523083 21745 ts_manager.cc:194] Registered new tserver with Master: 34e2861a29784992b9d1a9a5876087a6 (127.21.14.66:35979)
02:06:59.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.524083 21603 ts_manager.cc:194] Registered new tserver with Master: 34e2861a29784992b9d1a9a5876087a6 (127.21.14.66:35979)
02:06:59.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.525063 21603 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.21.14.66:45807
02:06:59.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:59.824193 22113 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:06:59.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:59.824496 22113 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:06:59.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:59.824602 22113 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:06:59.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:59.832774 22113 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:06:59.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:59.832947 22113 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.14.67
02:06:59.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.842504 22113 tablet_server_runner.cc:78] Tablet server non-default flags:
02:06:59.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:06:59.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.14.84:44189
02:06:59.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:06:59.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:06:59.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/data
02:06:59.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/wal
02:06:59.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:06:59.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:06:59.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.14.67:0
02:06:59.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:06:59.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:06:59.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:06:59.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/data/info.pb
02:06:59.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.14.67
02:06:59.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:06:59.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.21.14.126:34291,127.21.14.125:33787,127.21.14.124:36603
02:06:59.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:06:59.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:06:59.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:06:59.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:06:59.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:06:59.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.21.14.67
02:06:59.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/logs
02:06:59.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:06:59.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:06:59.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:06:59.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:06:59.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:06:59.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
02:06:59.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:06:59.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:43:30 UTC on bdcb31816ec0
02:06:59.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12766
02:06:59.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
02:06:59.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.844609 22113 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:06:59.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.846460 22113 file_cache.cc:504] Constructed file cache file cache with capacity 419430
02:06:59.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:59.855050 22118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:06:59.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:59.855110 22119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:06:59.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:06:59.856063 22121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:06:59.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.856405 22113 server_base.cc:1061] running on GCE node
02:06:59.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.856968 22113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:06:59.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.857833 22113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:06:59.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.859020 22113 hybrid_clock.cc:648] HybridClock initialized: now 1783476419858968 us; error 44 us; skew 500 ppm
02:06:59.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.859324 22113 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:06:59.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.861315 22113 webserver.cc:533] Webserver started at http://127.21.14.67:37049/ using document root <none> and password file <none>
02:06:59.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.862039 22113 fs_manager.cc:362] Metadata directory not provided
02:06:59.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.862145 22113 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:06:59.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.862519 22113 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
02:06:59.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.864831 22113 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/data/instance:
02:06:59.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "1a735d9d94bb4c2e8d28cad5e18d6785"
02:06:59.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:06:59 on dist-test-slave-545r"
02:06:59.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.865558 22113 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/wal/instance:
02:06:59.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "1a735d9d94bb4c2e8d28cad5e18d6785"
02:06:59.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:06:59 on dist-test-slave-545r"
02:06:59.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.870520 22113 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
02:06:59.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.873508 22127 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:06:59.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.874725 22113 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
02:06:59.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.874838 22113 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/data,/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/wal
02:06:59.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "1a735d9d94bb4c2e8d28cad5e18d6785"
02:06:59.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:06:59 on dist-test-slave-545r"
02:06:59.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.874986 22113 fs_report.cc:389] FS layout report
02:06:59.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:06:59.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/wal
02:06:59.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/wal
02:06:59.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/data/data
02:06:59.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:06:59.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:06:59.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:06:59.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:06:59.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:06:59.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:06:59.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:06:59.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:06:59.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:06:59.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:06:59.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:06:59.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.891921 22113 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:06:59.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.892650 22113 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:06:59.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.892850 22113 kserver.cc:163] Server-wide thread pool size limit: 3276
02:06:59.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.893622 22113 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:06:59.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.895100 22113 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
02:06:59.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.895181 22113 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
02:06:59.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.895246 22113 ts_tablet_manager.cc:616] Registered 0 tablets
02:06:59.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.895292 22113 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
02:06:59.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.924861 22113 rpc_server.cc:307] RPC server started. Bound to: 127.21.14.67:42423
02:06:59.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.924924 22239 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.14.67:42423 every 8 connection(s)
02:06:59.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.926137 22113 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/data/info.pb
02:06:59.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.936293 21561 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu as pid 22113
02:06:59.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.936442 21561 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/wal/instance
02:06:59.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.940788 22242 heartbeater.cc:344] Connected to a master server at 127.21.14.125:33787
02:06:59.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.940826 22240 heartbeater.cc:344] Connected to a master server at 127.21.14.124:36603
02:06:59.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.941169 22240 heartbeater.cc:461] Registering TS with master...
02:06:59.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.941174 22242 heartbeater.cc:461] Registering TS with master...
02:06:59.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.941231 22241 heartbeater.cc:344] Connected to a master server at 127.21.14.126:34291
02:06:59.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.942041 22241 heartbeater.cc:461] Registering TS with master...
02:06:59.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.942461 22242 heartbeater.cc:507] Master 127.21.14.125:33787 requested a full tablet report, sending...
02:06:59.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.942494 22240 heartbeater.cc:507] Master 127.21.14.124:36603 requested a full tablet report, sending...
02:06:59.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.942613 22241 heartbeater.cc:507] Master 127.21.14.126:34291 requested a full tablet report, sending...
02:06:59.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.944020 21603 ts_manager.cc:194] Registered new tserver with Master: 1a735d9d94bb4c2e8d28cad5e18d6785 (127.21.14.67:42423)
02:06:59.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.944245 21745 ts_manager.cc:194] Registered new tserver with Master: 1a735d9d94bb4c2e8d28cad5e18d6785 (127.21.14.67:42423)
02:06:59.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.944978 21673 ts_manager.cc:194] Registered new tserver with Master: 1a735d9d94bb4c2e8d28cad5e18d6785 (127.21.14.67:42423)
02:06:59.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.945076 21603 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.21.14.67:57389
02:06:59.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:06:59.959007 21561 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
02:06:59.961 [DEBUG - main] (MiniKuduCluster.java:195) Response: 
02:06:59.962 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_masters {
}

02:06:59.970 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_masters {
  masters {
    id {
      type: MASTER
      index: 0
    }
    bound_rpc_address {
      host: "127.21.14.126"
      port: 34291
    }
    bound_http_address {
      host: "127.21.14.126"
      port: 36679
    }
  }
  masters {
    id {
      type: MASTER
      index: 1
    }
    bound_rpc_address {
      host: "127.21.14.125"
      port: 33787
    }
    bound_http_address {
      host: "127.21.14.125"
      port: 35241
    }
  }
  masters {
    id {
      type: MASTER
      index: 2
    }
    bound_rpc_address {
      host: "127.21.14.124"
      port: 36603
    }
    bound_http_address {
      host: "127.21.14.124"
      port: 46349
    }
  }
}

02:06:59.974 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_tservers {
}

02:06:59.976 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_tservers {
  tservers {
    id {
      type: TSERVER
      index: 0
    }
    bound_rpc_address {
      host: "127.21.14.65"
      port: 38711
    }
    bound_http_address {
      host: "127.21.14.65"
      port: 36395
    }
  }
  tservers {
    id {
      type: TSERVER
      index: 1
    }
    bound_rpc_address {
      host: "127.21.14.66"
      port: 35979
    }
    bound_http_address {
      host: "127.21.14.66"
      port: 43569
    }
  }
  tservers {
    id {
      type: TSERVER
      index: 2
    }
    bound_rpc_address {
      host: "127.21.14.67"
      port: 42423
    }
    bound_http_address {
      host: "127.21.14.67"
      port: 37049
    }
  }
}

02:06:59.977 [INFO - main] (KuduTestHarness.java:157) Creating a new Kudu client...
02:07:00.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:00.106832 21970 heartbeater.cc:499] Master 127.21.14.126:34291 was elected leader, sending a full tablet report...
02:07:00.134 [DEBUG - main] (CompositeStrategy.java:59) Unable to load modern Subject methods, relying only on legacy methods
java.lang.NoSuchMethodException: javax.security.auth.Subject.current()
	at java.base/java.lang.Class.getDeclaredMethod(Class.java:2677)
	at org.apache.kudu.client.internals.ModernStrategy.<init>(ModernStrategy.java:51)
	at org.apache.kudu.client.internals.CompositeStrategy.<init>(CompositeStrategy.java:52)
	at org.apache.kudu.client.internals.CompositeStrategy.<clinit>(CompositeStrategy.java:41)
	at org.apache.kudu.client.internals.SecurityManagerCompatibility.get(SecurityManagerCompatibility.java:43)
	at org.apache.kudu.client.SecurityContext.setupSubject(SecurityContext.java:155)
	at org.apache.kudu.client.SecurityContext.<init>(SecurityContext.java:138)
	at org.apache.kudu.client.AsyncKuduClient.<init>(AsyncKuduClient.java:431)
	at org.apache.kudu.client.AsyncKuduClient$AsyncKuduClientBuilder.build(AsyncKuduClient.java:3109)
	at org.apache.kudu.test.KuduTestHarness.before(KuduTestHarness.java:161)
	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:50)
	at org.apache.kudu.test.junit.RetryRule$RetryStatement.doOneAttemptAndReport(RetryRule.java:184)
	at org.apache.kudu.test.junit.RetryRule$RetryStatement.evaluate(RetryRule.java:244)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.junit.runners.Suite.runChild(Suite.java:128)
	at org.junit.runners.Suite.runChild(Suite.java:27)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
	at org.junit.runner.JUnitCore.runMain(JUnitCore.java:77)
	at org.junit.runner.JUnitCore.main(JUnitCore.java:36)
02:07:00.190 [DEBUG - main] (SecurityUtil.java:122) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 
02:07:00.331 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.21.14.126:34291(127.21.14.126:34291)] connecting to peer
02:07:00.375 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.21.14.124:36603(127.21.14.124:36603)] connecting to peer
02:07:00.377 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.21.14.125:33787(127.21.14.125:33787)] connecting to peer
02:07:00.395 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.21.14.126:34291(127.21.14.126:34291)] Successfully connected to peer
02:07:00.395 [DEBUG - kudu-nio-2] (Connection.java:757) [peer master-127.21.14.125:33787(127.21.14.125:33787)] Successfully connected to peer
02:07:00.395 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.21.14.124:36603(127.21.14.124:36603)] Successfully connected to peer
02:07:00.485 [DEBUG - kudu-nio-2] (Negotiator.java:528) SASL mechanism PLAIN chosen for peer 127.21.14.125
02:07:00.489 [DEBUG - kudu-nio-0] (Negotiator.java:528) SASL mechanism PLAIN chosen for peer 127.21.14.126
02:07:00.489 [DEBUG - kudu-nio-1] (Negotiator.java:528) SASL mechanism PLAIN chosen for peer 127.21.14.124
02:07:00.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:00.526932 22107 heartbeater.cc:499] Master 127.21.14.126:34291 was elected leader, sending a full tablet report...
02:07:00.691 [DEBUG - kudu-nio-1] (Negotiator.java:918) Authenticated connection [id: 0x7f2bdfb1, L:/127.0.0.1:38036 - R:/127.21.14.124:36603] using SASL/PLAIN
02:07:00.691 [DEBUG - kudu-nio-0] (Negotiator.java:918) Authenticated connection [id: 0x72e23b40, L:/127.0.0.1:33088 - R:/127.21.14.126:34291] using SASL/PLAIN
02:07:00.693 [DEBUG - kudu-nio-2] (Negotiator.java:918) Authenticated connection [id: 0x503f30f9, L:/127.0.0.1:47680 - R:/127.21.14.125:33787] using SASL/PLAIN
02:07:00.727 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet Kudu Master for table 'Kudu Master' with partition [<start>, <end>)
02:07:00.727 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=CreateTable, tablet=null, attempt=1, TimeoutTracker(timeout=50000, elapsed=432), Traces: [0ms] refreshing cache from master, [24ms] Sub RPC ConnectToMaster: sending RPC to server master-127.21.14.126:34291, [77ms] Sub RPC ConnectToMaster: sending RPC to server master-127.21.14.124:36603, [79ms] Sub RPC ConnectToMaster: sending RPC to server master-127.21.14.125:33787, [405ms] Sub RPC ConnectToMaster: received response from server master-127.21.14.124:36603: OK, [405ms] Sub RPC ConnectToMaster: received response from server master-127.21.14.125:33787: OK, [406ms] Sub RPC ConnectToMaster: received response from server master-127.21.14.126:34291: OK, deferred=null) after lookup
02:07:00.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:00.768185 21603 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33088:
02:07:00.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) name: "org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283"
02:07:00.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) schema {
02:07:00.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
02:07:00.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "key"
02:07:00.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
02:07:00.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: true
02:07:00.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
02:07:00.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
02:07:00.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
02:07:00.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
02:07:00.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
02:07:00.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
02:07:00.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column1_i"
02:07:00.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
02:07:00.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
02:07:00.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
02:07:00.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
02:07:00.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
02:07:00.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
02:07:00.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
02:07:00.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
02:07:00.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column2_i"
02:07:00.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
02:07:00.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
02:07:00.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
02:07:00.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
02:07:00.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
02:07:00.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
02:07:00.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
02:07:00.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
02:07:00.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column3_s"
02:07:00.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: STRING
02:07:00.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
02:07:00.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: true
02:07:00.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     encoding: DICT_ENCODING
02:07:00.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     compression: LZ4
02:07:00.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 4096
02:07:00.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
02:07:00.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
02:07:00.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
02:07:00.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
02:07:00.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column4_b"
02:07:00.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: BOOL
02:07:00.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
02:07:00.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
02:07:00.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
02:07:00.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
02:07:00.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
02:07:00.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
02:07:00.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) }
02:07:00.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) partition_schema {
02:07:00.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   range_schema {
02:07:00.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     columns {
02:07:00.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)       name: "key"
02:07:00.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     }
02:07:00.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
02:07:00.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) }
02:07:00.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:00.770293 21603 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
02:07:00.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:00.845685 22175 tablet_service.cc:1511] Processing CreateTablet for tablet 5637f0c18ff54bdd81b35dab09fada4d (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283 [id=a70d97e0192b409da00a559612d8aceb]), partition=RANGE (key) PARTITION UNBOUNDED
02:07:00.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:00.847618 22175 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5637f0c18ff54bdd81b35dab09fada4d. 1 dirs total, 0 dirs full, 0 dirs failed
02:07:00.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:00.851495 22041 tablet_service.cc:1511] Processing CreateTablet for tablet 5637f0c18ff54bdd81b35dab09fada4d (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283 [id=a70d97e0192b409da00a559612d8aceb]), partition=RANGE (key) PARTITION UNBOUNDED
02:07:00.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:00.852782 22041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5637f0c18ff54bdd81b35dab09fada4d. 1 dirs total, 0 dirs full, 0 dirs failed
02:07:00.868 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:894) Opened table a70d97e0192b409da00a559612d8aceb
02:07:00.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:00.873543 22269 tablet_bootstrap.cc:492] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785: Bootstrap starting.
02:07:00.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:00.876636 21904 tablet_service.cc:1511] Processing CreateTablet for tablet 5637f0c18ff54bdd81b35dab09fada4d (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283 [id=a70d97e0192b409da00a559612d8aceb]), partition=RANGE (key) PARTITION UNBOUNDED
02:07:00.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:00.877651 22268 tablet_bootstrap.cc:492] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6: Bootstrap starting.
02:07:00.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:00.878712 21904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5637f0c18ff54bdd81b35dab09fada4d. 1 dirs total, 0 dirs full, 0 dirs failed
02:07:00.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:00.879701 22269 tablet_bootstrap.cc:654] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785: Neither blocks nor log segments found. Creating new log.
02:07:00.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:00.882843 22268 tablet_bootstrap.cc:654] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6: Neither blocks nor log segments found. Creating new log.
02:07:00.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:00.884047 22268 log.cc:826] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6: Log is configured to *not* fsync() on all Append() calls
02:07:00.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:00.885480 22269 log.cc:826] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785: Log is configured to *not* fsync() on all Append() calls
02:07:00.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:00.897397 22268 tablet_bootstrap.cc:492] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6: No bootstrap required, opened a new log
02:07:00.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:00.897398 22269 tablet_bootstrap.cc:492] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785: No bootstrap required, opened a new log
02:07:00.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:00.897696 22268 ts_tablet_manager.cc:1403] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6: Time spent bootstrapping tablet: real 0.020s	user 0.003s	sys 0.008s
02:07:00.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:00.897699 22269 ts_tablet_manager.cc:1403] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785: Time spent bootstrapping tablet: real 0.024s	user 0.011s	sys 0.001s
02:07:00.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:00.904700 22268 raft_consensus.cc:359] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:00.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:00.904706 22269 raft_consensus.cc:359] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:00.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:00.905078 22269 raft_consensus.cc:385] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
02:07:00.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:00.905078 22268 raft_consensus.cc:385] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
02:07:00.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:00.905176 22268 raft_consensus.cc:740] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34e2861a29784992b9d1a9a5876087a6, State: Initialized, Role: FOLLOWER
02:07:00.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:00.905176 22269 raft_consensus.cc:740] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a735d9d94bb4c2e8d28cad5e18d6785, State: Initialized, Role: FOLLOWER
02:07:00.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:00.905844 22268 consensus_queue.cc:260] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:00.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:00.905844 22269 consensus_queue.cc:260] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:00.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:00.908771 22272 tablet_bootstrap.cc:492] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af: Bootstrap starting.
02:07:00.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:00.912505 22268 ts_tablet_manager.cc:1434] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6: Time spent starting tablet: real 0.015s	user 0.008s	sys 0.003s
02:07:00.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:00.913700 22272 tablet_bootstrap.cc:654] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af: Neither blocks nor log segments found. Creating new log.
02:07:00.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:00.917743 22241 heartbeater.cc:499] Master 127.21.14.126:34291 was elected leader, sending a full tablet report...
02:07:00.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:00.918552 22269 ts_tablet_manager.cc:1434] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785: Time spent starting tablet: real 0.021s	user 0.007s	sys 0.008s
02:07:00.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:00.922508 22272 log.cc:826] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af: Log is configured to *not* fsync() on all Append() calls
02:07:00.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:00.928416 22272 tablet_bootstrap.cc:492] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af: No bootstrap required, opened a new log
02:07:00.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:00.928721 22272 ts_tablet_manager.cc:1403] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af: Time spent bootstrapping tablet: real 0.020s	user 0.005s	sys 0.006s
02:07:00.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:00.929324 22243 tablet.cc:2406] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785: Can't schedule compaction. Clean time has not been advanced past its initial value.
02:07:00.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:00.937561 22272 raft_consensus.cc:359] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:00.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:00.938477 22272 raft_consensus.cc:385] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
02:07:00.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:00.938753 22272 raft_consensus.cc:740] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 790137564e884f989bbd5c36475181af, State: Initialized, Role: FOLLOWER
02:07:00.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:00.939667 22272 consensus_queue.cc:260] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:00.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:00.946425 22272 ts_tablet_manager.cc:1434] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af: Time spent starting tablet: real 0.017s	user 0.005s	sys 0.007s
02:07:01.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:01.003626 22274 raft_consensus.cc:493] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
02:07:01.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:01.003927 22274 raft_consensus.cc:515] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:01.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:01.006193 22274 leader_election.cc:290] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 34e2861a29784992b9d1a9a5876087a6 (127.21.14.66:35979), 790137564e884f989bbd5c36475181af (127.21.14.65:38711)
02:07:01.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:01.006227 22109 tablet.cc:2406] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
02:07:01.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:01.014463 22061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5637f0c18ff54bdd81b35dab09fada4d" candidate_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34e2861a29784992b9d1a9a5876087a6" is_pre_election: true
02:07:01.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:01.014761 21924 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5637f0c18ff54bdd81b35dab09fada4d" candidate_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "790137564e884f989bbd5c36475181af" is_pre_election: true
02:07:01.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:01.014987 22061 raft_consensus.cc:2468] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a735d9d94bb4c2e8d28cad5e18d6785 in term 0.
02:07:01.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:01.015267 21924 raft_consensus.cc:2468] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a735d9d94bb4c2e8d28cad5e18d6785 in term 0.
02:07:01.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:01.015646 22128 leader_election.cc:304] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a735d9d94bb4c2e8d28cad5e18d6785, 34e2861a29784992b9d1a9a5876087a6; no voters: 
02:07:01.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:01.016022 22274 raft_consensus.cc:2804] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [term 0 FOLLOWER]: Leader pre-election won for term 1
02:07:01.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:01.016149 22274 raft_consensus.cc:493] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
02:07:01.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:01.016223 22274 raft_consensus.cc:3060] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [term 0 FOLLOWER]: Advancing to term 1
02:07:01.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:01.018218 22274 raft_consensus.cc:515] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:01.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:01.018774 22274 leader_election.cc:290] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [CANDIDATE]: Term 1 election: Requested vote from peers 34e2861a29784992b9d1a9a5876087a6 (127.21.14.66:35979), 790137564e884f989bbd5c36475181af (127.21.14.65:38711)
02:07:01.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:01.019182 22061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5637f0c18ff54bdd81b35dab09fada4d" candidate_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34e2861a29784992b9d1a9a5876087a6"
02:07:01.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:01.019217 21924 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5637f0c18ff54bdd81b35dab09fada4d" candidate_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "790137564e884f989bbd5c36475181af"
02:07:01.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:01.019359 22061 raft_consensus.cc:3060] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [term 0 FOLLOWER]: Advancing to term 1
02:07:01.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:01.019366 21924 raft_consensus.cc:3060] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 0 FOLLOWER]: Advancing to term 1
02:07:01.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:01.021718 21924 raft_consensus.cc:2468] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a735d9d94bb4c2e8d28cad5e18d6785 in term 1.
02:07:01.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:01.022133 22129 leader_election.cc:304] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a735d9d94bb4c2e8d28cad5e18d6785, 790137564e884f989bbd5c36475181af; no voters: 
02:07:01.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:01.022405 22274 raft_consensus.cc:2804] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [term 1 FOLLOWER]: Leader election won for term 1
02:07:01.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:01.022464 22061 raft_consensus.cc:2468] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a735d9d94bb4c2e8d28cad5e18d6785 in term 1.
02:07:01.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:01.023272 22274 raft_consensus.cc:697] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [term 1 LEADER]: Becoming Leader. State: Replica: 1a735d9d94bb4c2e8d28cad5e18d6785, State: Running, Role: LEADER
02:07:01.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:01.023806 22274 consensus_queue.cc:237] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:01.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:01.029124 21603 catalog_manager.cc:5697] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a735d9d94bb4c2e8d28cad5e18d6785 (127.21.14.67). New cstate: current_term: 1 leader_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } health_report { overall_health: UNKNOWN } } }
02:07:01.042 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:894) Opened table a70d97e0192b409da00a559612d8aceb
02:07:01.058 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:01.065 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299993, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)[L],34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979),790137564e884f989bbd5c36475181af(127.21.14.65:38711)]}]
02:07:01.066 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=null, attempt=1, sequence_id=-1, TimeoutTracker(timeout=30000, elapsed=17), Traces: [0ms] refreshing cache from master, [0ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [7ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=null) after lookup
02:07:01.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:01.078051 21972 tablet.cc:2406] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af: Can't schedule compaction. Clean time has not been advanced past its initial value.
02:07:01.078 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)] connecting to peer
02:07:01.080 [DEBUG - kudu-nio-3] (Connection.java:757) [peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)] Successfully connected to peer
02:07:01.159 [DEBUG - kudu-nio-3] (Negotiator.java:918) Authenticated connection [id: 0xa9e017ba, L:/127.0.0.1:50742 - R:/127.21.14.67:42423] using TOKEN/null
02:07:01.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:01.171756 22061 raft_consensus.cc:1275] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [term 1 FOLLOWER]: Refusing update from remote peer 1a735d9d94bb4c2e8d28cad5e18d6785: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
02:07:01.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:01.171754 21924 raft_consensus.cc:1275] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 1 FOLLOWER]: Refusing update from remote peer 1a735d9d94bb4c2e8d28cad5e18d6785: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
02:07:01.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:01.172981 22279 consensus_queue.cc:1048] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [LEADER]: Connected to new peer: Peer: permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
02:07:01.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:01.173272 22274 consensus_queue.cc:1048] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
02:07:01.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:01.191156 22282 mvcc.cc:204] Tried to move back new op lower bound from 7305119421107650560 to 7305119420517527552. Current Snapshot: MvccSnapshot[applied={T|T < 7305119421107650560}]
02:07:01.309 [DEBUG - kudu-nio-3] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 3 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283, tablet=5637f0c18ff54bdd81b35dab09fada4d, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:07:01.311 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.21.14.67:42423
02:07:01.312 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

02:07:01.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:01.313342 21561 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu pid 22113
02:07:01.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:01.332979 22235 generic_service.cc:196] Checking for leaks (request via RPC)
02:07:01.435 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:01.435387 21561 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu with pid 22113
02:07:01.453 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:07:01.456 [DEBUG - kudu-nio-3] (Connection.java:239) [peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)] handling channelInactive
02:07:01.457 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)] cleaning up while in state READY due to: connection closed
02:07:01.457 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423) for tablet 5637f0c18ff54bdd81b35dab09fada4d: connection closed
02:07:01.478 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:01.478 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)[L],34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979),790137564e884f989bbd5c36475181af(127.21.14.65:38711)]}]
02:07:01.479 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5637f0c18ff54bdd81b35dab09fada4d, attempt=2, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=25), Traces: [0ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [4ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: connection closed, [5ms] delaying RPC due to: Network error: connection closed, [22ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [25ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=Deferred@1185479999(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007fe67c338998@326aada6 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6d56116 -> wakeup thread Time-limited test)) after lookup
02:07:01.480 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)] connecting to peer
02:07:01.482 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:01.482 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:01.482 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423) for tablet 5637f0c18ff54bdd81b35dab09fada4d: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:01.498 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:01.498 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)[L],34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979),790137564e884f989bbd5c36475181af(127.21.14.65:38711)]}]
02:07:01.499 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5637f0c18ff54bdd81b35dab09fada4d, attempt=4, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=45), Traces: [0ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [4ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: connection closed, [5ms] delaying RPC due to: Network error: connection closed, [22ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [25ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [26ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [29ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [29ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [43ms] refreshing cache from master, [43ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [45ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=Deferred@1185479999(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007fe67c338998@326aada6 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@962150028 after retry RPC@68845974), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6d56116 -> wakeup thread Time-limited test -> (continuation of Deferred@962150028 after retry RPC@68845974))) after lookup
02:07:01.499 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)] connecting to peer
02:07:01.501 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:01.501 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:01.501 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423) for tablet 5637f0c18ff54bdd81b35dab09fada4d: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:01.518 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:01.518 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)[L],34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979),790137564e884f989bbd5c36475181af(127.21.14.65:38711)]}]
02:07:01.519 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5637f0c18ff54bdd81b35dab09fada4d, attempt=6, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=65), Traces: [0ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [4ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: connection closed, [5ms] delaying RPC due to: Network error: connection closed, [22ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [25ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [26ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [29ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [29ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [43ms] refreshing cache from master, [43ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [45ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [46ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [48ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [48ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [62ms] refreshing cache from master, [62ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [64ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=Deferred@1185479999(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007fe67c338998@326aada6 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@962150028 after retry RPC@68845974) -> (continuation of Deferred@1201707748 after retry RPC@878468493), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6d56116 -> wakeup thread Time-limited test -> (continuation of Deferred@962150028 after retry RPC@68845974) -> (continuation of Deferred@1201707748 after retry RPC@878468493))) after lookup
02:07:01.519 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)] connecting to peer
02:07:01.521 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:01.521 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:01.521 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423) for tablet 5637f0c18ff54bdd81b35dab09fada4d: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:01.619 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:01.620 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)[L],34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979),790137564e884f989bbd5c36475181af(127.21.14.65:38711)]}]
02:07:01.620 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5637f0c18ff54bdd81b35dab09fada4d, attempt=8, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=166), Traces: [0ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [4ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: connection closed, [5ms] delaying RPC due to: Network error: connection closed, [22ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [25ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [26ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [29ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [29ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [43ms] refreshing cache from master, [43ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [45ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [46ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [48ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [48ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [62ms] refreshing cache from master, [62ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [64ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [66ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [68ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [68ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [163ms] refreshing cache from master, [163ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [166ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=Deferred@1185479999(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007fe67c338998@326aada6 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@962150028 after retry RPC@68845974) -> (continuation of Deferred@1201707748 after retry RPC@878468493) -> (continuation of Deferred@1043720655 after retry RPC@1144324113), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6d56116 -> wakeup thread Time-limited test -> (continuation of Deferred@962150028 after retry RPC@68845974) -> (continuation of Deferred@1201707748 after retry RPC@878468493) -> (continuation of Deferred@1043720655 after retry RPC@1144324113))) after lookup
02:07:01.621 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)] connecting to peer
02:07:01.622 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:01.622 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:01.622 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423) for tablet 5637f0c18ff54bdd81b35dab09fada4d: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:01.999 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:02.000 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)[L],34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979),790137564e884f989bbd5c36475181af(127.21.14.65:38711)]}]
02:07:02.002 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5637f0c18ff54bdd81b35dab09fada4d, attempt=10, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=548), Traces: [0ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [4ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: connection closed, [5ms] delaying RPC due to: Network error: connection closed, [22ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [25ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [26ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [29ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [29ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [43ms] refreshing cache from master, [43ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [45ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [46ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [48ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [48ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [62ms] refreshing cache from master, [62ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [64ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [66ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [68ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [68ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [163ms] refreshing cache from master, [163ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [166ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [167ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [169ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [170ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [543ms] refreshing cache from master, [543ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [546ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=Deferred@1185479999(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007fe67c338998@326aada6 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@962150028 after retry RPC@68845974) -> (continuation of Deferred@1201707748 after retry RPC@878468493) -> (continuation of Deferred@1043720655 after retry RPC@1144324113) -> (continuation of Deferred@1553398342 after retry RPC@2031712225), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6d56116 -> wakeup thread Time-limited test -> (continuation of Deferred@962150028 after retry RPC@68845974) -> (continuation of Deferred@1201707748 after retry RPC@878468493) -> (continuation of Deferred@1043720655 after retry RPC@1144324113) -> (continuation of Deferred@1553398342 after retry RPC@2031712225))) after lookup
02:07:02.003 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)] connecting to peer
02:07:02.004 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:02.004 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:02.004 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423) for tablet 5637f0c18ff54bdd81b35dab09fada4d: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:02.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:02.751726 22298 raft_consensus.cc:493] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1a735d9d94bb4c2e8d28cad5e18d6785)
02:07:02.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:02.751906 22298 raft_consensus.cc:515] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:02.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:02.752775 22299 raft_consensus.cc:493] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1a735d9d94bb4c2e8d28cad5e18d6785)
02:07:02.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:02.752975 22299 raft_consensus.cc:515] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:02.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:02.753738 22298 leader_election.cc:290] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1a735d9d94bb4c2e8d28cad5e18d6785 (127.21.14.67:42423), 34e2861a29784992b9d1a9a5876087a6 (127.21.14.66:35979)
02:07:02.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:02.756145 21858 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.14.67:42423: connect: Connection refused (error 111)
02:07:02.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:02.760668 22299 leader_election.cc:290] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1a735d9d94bb4c2e8d28cad5e18d6785 (127.21.14.67:42423), 790137564e884f989bbd5c36475181af (127.21.14.65:38711)
02:07:02.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:02.768079 21858 leader_election.cc:336] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1a735d9d94bb4c2e8d28cad5e18d6785 (127.21.14.67:42423): Network error: Client connection negotiation failed: client connection to 127.21.14.67:42423: connect: Connection refused (error 111)
02:07:02.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:02.768883 21995 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.14.67:42423: connect: Connection refused (error 111)
02:07:02.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:02.771487 22061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5637f0c18ff54bdd81b35dab09fada4d" candidate_uuid: "790137564e884f989bbd5c36475181af" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "34e2861a29784992b9d1a9a5876087a6" is_pre_election: true
02:07:02.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:02.771716 22061 raft_consensus.cc:2468] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 790137564e884f989bbd5c36475181af in term 1.
02:07:02.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:02.772029 21995 leader_election.cc:336] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1a735d9d94bb4c2e8d28cad5e18d6785 (127.21.14.67:42423): Network error: Client connection negotiation failed: client connection to 127.21.14.67:42423: connect: Connection refused (error 111)
02:07:02.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:02.772192 21857 leader_election.cc:304] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 34e2861a29784992b9d1a9a5876087a6, 790137564e884f989bbd5c36475181af; no voters: 1a735d9d94bb4c2e8d28cad5e18d6785
02:07:02.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:02.772591 22298 raft_consensus.cc:2804] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 1 FOLLOWER]: Leader pre-election won for term 2
02:07:02.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:02.772722 22298 raft_consensus.cc:493] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1a735d9d94bb4c2e8d28cad5e18d6785)
02:07:02.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:02.772809 22298 raft_consensus.cc:3060] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 1 FOLLOWER]: Advancing to term 2
02:07:02.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:02.774933 21924 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5637f0c18ff54bdd81b35dab09fada4d" candidate_uuid: "34e2861a29784992b9d1a9a5876087a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "790137564e884f989bbd5c36475181af" is_pre_election: true
02:07:02.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:02.775100 22298 raft_consensus.cc:515] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:02.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:02.775449 21924 raft_consensus.cc:2393] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 34e2861a29784992b9d1a9a5876087a6 in current term 2: Already voted for candidate 790137564e884f989bbd5c36475181af in this term.
02:07:02.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:02.775790 22298 leader_election.cc:290] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [CANDIDATE]: Term 2 election: Requested vote from peers 1a735d9d94bb4c2e8d28cad5e18d6785 (127.21.14.67:42423), 34e2861a29784992b9d1a9a5876087a6 (127.21.14.66:35979)
02:07:02.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:02.776343 22061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5637f0c18ff54bdd81b35dab09fada4d" candidate_uuid: "790137564e884f989bbd5c36475181af" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "34e2861a29784992b9d1a9a5876087a6"
02:07:02.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:02.776584 22061 raft_consensus.cc:3060] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [term 1 FOLLOWER]: Advancing to term 2
02:07:02.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:02.776994 21858 leader_election.cc:336] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1a735d9d94bb4c2e8d28cad5e18d6785 (127.21.14.67:42423): Network error: Client connection negotiation failed: client connection to 127.21.14.67:42423: connect: Connection refused (error 111)
02:07:02.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:02.778592 22061 raft_consensus.cc:2468] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 790137564e884f989bbd5c36475181af in term 2.
02:07:02.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:02.778952 21857 leader_election.cc:304] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 34e2861a29784992b9d1a9a5876087a6, 790137564e884f989bbd5c36475181af; no voters: 1a735d9d94bb4c2e8d28cad5e18d6785
02:07:02.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:02.776322 21995 leader_election.cc:304] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 34e2861a29784992b9d1a9a5876087a6; no voters: 1a735d9d94bb4c2e8d28cad5e18d6785, 790137564e884f989bbd5c36475181af
02:07:02.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:02.779264 22299 raft_consensus.cc:2749] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
02:07:02.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:02.779364 22298 raft_consensus.cc:2804] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 2 FOLLOWER]: Leader election won for term 2
02:07:02.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:02.779903 22298 raft_consensus.cc:697] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 2 LEADER]: Becoming Leader. State: Replica: 790137564e884f989bbd5c36475181af, State: Running, Role: LEADER
02:07:02.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:02.780398 22298 consensus_queue.cc:237] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:02.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:02.784107 21603 catalog_manager.cc:5697] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af reported cstate change: term changed from 1 to 2, leader changed from 1a735d9d94bb4c2e8d28cad5e18d6785 (127.21.14.67) to 790137564e884f989bbd5c36475181af (127.21.14.65). New cstate: current_term: 2 leader_uuid: "790137564e884f989bbd5c36475181af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } health_report { overall_health: HEALTHY } } }
02:07:03.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:03.181584 22061 raft_consensus.cc:1275] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [term 2 FOLLOWER]: Refusing update from remote peer 790137564e884f989bbd5c36475181af: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
02:07:03.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:03.182324 22298 consensus_queue.cc:1048] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [LEADER]: Connected to new peer: Peer: permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
02:07:03.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:03.188051 21858 consensus_peers.cc:597] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af -> Peer 1a735d9d94bb4c2e8d28cad5e18d6785 (127.21.14.67:42423): Couldn't send request to peer 1a735d9d94bb4c2e8d28cad5e18d6785. Status: Network error: Client connection negotiation failed: client connection to 127.21.14.67:42423: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
02:07:03.299 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:03.299 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423),34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979),790137564e884f989bbd5c36475181af(127.21.14.65:38711)[L]]}]
02:07:03.300 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5637f0c18ff54bdd81b35dab09fada4d, attempt=12, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=1846), Traces: [0ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [4ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: connection closed, [5ms] delaying RPC due to: Network error: connection closed, [22ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [25ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [26ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [29ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [29ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [43ms] refreshing cache from master, [43ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [45ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [46ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [48ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [48ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [62ms] refreshing cache from master, [62ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [64ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [66ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [68ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [68ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [163ms] refreshing cache from master, [163ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [166ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [167ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [169ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [170ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [543ms] refreshing cache from master, [543ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [546ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [550ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [551ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [551ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [1842ms] refreshing cache from master, [1843ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [1845ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=Deferred@1185479999(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007fe67c338998@326aada6 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@962150028 after retry RPC@68845974) -> (continuation of Deferred@1201707748 after retry RPC@878468493) -> (continuation of Deferred@1043720655 after retry RPC@1144324113) -> (continuation of Deferred@1553398342 after retry RPC@2031712225) -> (continuation of Deferred@2018721985 after retry RPC@1964322125), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6d56116 -> wakeup thread Time-limited test -> (continuation of Deferred@962150028 after retry RPC@68845974) -> (continuation of Deferred@1201707748 after retry RPC@878468493) -> (continuation of Deferred@1043720655 after retry RPC@1144324113) -> (continuation of Deferred@1553398342 after retry RPC@2031712225) -> (continuation of Deferred@2018721985 after retry RPC@1964322125))) after lookup
02:07:03.301 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711)] connecting to peer
02:07:03.302 [DEBUG - kudu-nio-1] (Connection.java:757) [peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711)] Successfully connected to peer
02:07:03.322 [DEBUG - kudu-nio-1] (Negotiator.java:918) Authenticated connection [id: 0x3cfb1a43, L:/127.0.0.1:37134 - R:/127.21.14.65:38711] using TOKEN/null
02:07:03.342 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.21.14.67:42423
02:07:03.343 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

02:07:03.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:03.344571 21561 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu
02:07:03.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu
02:07:03.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/wal
02:07:03.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/data
02:07:03.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:07:03.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:07:03.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:07:03.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:07:03.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:07:03.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:07:03.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/logs
02:07:03.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/data/info.pb
02:07:03.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:07:03.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:07:03.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:07:03.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:07:03.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:07:03.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:07:03.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:07:03.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:07:03.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:07:03.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:07:03.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.14.67:42423
02:07:03.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.21.14.67
02:07:03.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.21.14.126:34291,127.21.14.125:33787,127.21.14.124:36603
02:07:03.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=37049
02:07:03.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.14.67
02:07:03.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.14.84:44189
02:07:03.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:07:03.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:07:03.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:07:03.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:03.648739 22327 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:07:03.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:03.648983 22327 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:07:03.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:03.649063 22327 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:07:03.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:03.656960 22327 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:07:03.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:03.657090 22327 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.14.67
02:07:03.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:03.666332 22327 tablet_server_runner.cc:78] Tablet server non-default flags:
02:07:03.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:07:03.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.14.84:44189
02:07:03.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:07:03.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:07:03.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/data
02:07:03.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/wal
02:07:03.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:07:03.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:07:03.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.14.67:42423
02:07:03.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:07:03.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:07:03.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:07:03.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/data/info.pb
02:07:03.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.14.67
02:07:03.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=37049
02:07:03.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.21.14.126:34291,127.21.14.125:33787,127.21.14.124:36603
02:07:03.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:07:03.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:07:03.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:07:03.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:07:03.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:07:03.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.21.14.67
02:07:03.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/logs
02:07:03.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:07:03.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:07:03.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:07:03.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:07:03.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:07:03.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
02:07:03.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:07:03.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:43:30 UTC on bdcb31816ec0
02:07:03.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12766
02:07:03.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
02:07:03.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:03.667907 22327 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:07:03.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:03.669524 22327 file_cache.cc:504] Constructed file cache file cache with capacity 419430
02:07:03.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:03.678349 22333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:03.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:03.678383 22332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:03.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:03.678846 22335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:03.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:03.679286 22327 server_base.cc:1061] running on GCE node
02:07:03.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:03.679795 22327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:07:03.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:03.680655 22327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:07:03.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:03.681849 22327 hybrid_clock.cc:648] HybridClock initialized: now 1783476423681809 us; error 29 us; skew 500 ppm
02:07:03.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:03.682157 22327 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:07:03.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:03.684098 22327 webserver.cc:533] Webserver started at http://127.21.14.67:37049/ using document root <none> and password file <none>
02:07:03.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:03.684825 22327 fs_manager.cc:362] Metadata directory not provided
02:07:03.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:03.684937 22327 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:07:03.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:03.689832 22327 fs_manager.cc:714] Time spent opening directory manager: real 0.003s	user 0.000s	sys 0.004s
02:07:03.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:03.692287 22341 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:07:03.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:03.693446 22327 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.000s	sys 0.003s
02:07:03.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:03.693588 22327 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/data,/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/wal
02:07:03.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "1a735d9d94bb4c2e8d28cad5e18d6785"
02:07:03.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:06:59 on dist-test-slave-545r"
02:07:03.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:03.694242 22327 fs_report.cc:389] FS layout report
02:07:03.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:07:03.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/wal
02:07:03.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/wal
02:07:03.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/data/data
02:07:03.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:07:03.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:07:03.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:07:03.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:07:03.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:07:03.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:07:03.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:07:03.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:07:03.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:07:03.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:07:03.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:07:03.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:03.719770 22327 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:07:03.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:03.720525 22327 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:07:03.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:03.720803 22327 kserver.cc:163] Server-wide thread pool size limit: 3276
02:07:03.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:03.721680 22327 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:07:03.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:03.723588 22348 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
02:07:03.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:03.730700 22327 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
02:07:03.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:03.730780 22327 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.008s	user 0.001s	sys 0.000s
02:07:03.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:03.730899 22327 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
02:07:03.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:03.733765 22327 ts_tablet_manager.cc:616] Registered 1 tablets
02:07:03.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:03.733855 22327 ts_tablet_manager.cc:595] Time spent register tablets: real 0.003s	user 0.002s	sys 0.000s
02:07:03.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:03.734243 22348 tablet_bootstrap.cc:492] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785: Bootstrap starting.
02:07:03.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:03.753904 22348 log.cc:826] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785: Log is configured to *not* fsync() on all Append() calls
02:07:03.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:03.766010 22348 tablet_bootstrap.cc:492] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
02:07:03.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:03.766927 22348 tablet_bootstrap.cc:492] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785: Bootstrap complete.
02:07:03.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:03.768345 22348 ts_tablet_manager.cc:1403] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785: Time spent bootstrapping tablet: real 0.034s	user 0.020s	sys 0.012s
02:07:03.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:03.771299 22327 rpc_server.cc:307] RPC server started. Bound to: 127.21.14.67:42423
02:07:03.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:03.771395 22455 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.14.67:42423 every 8 connection(s)
02:07:03.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:03.773984 22327 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/data/info.pb
02:07:03.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:03.773723 22348 raft_consensus.cc:359] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:03.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:03.774565 22348 raft_consensus.cc:740] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a735d9d94bb4c2e8d28cad5e18d6785, State: Initialized, Role: FOLLOWER
02:07:03.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:03.775583 22348 consensus_queue.cc:260] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:03.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:03.778604 21561 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu as pid 22327
02:07:03.779 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:07:03.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:03.789438 22348 ts_tablet_manager.cc:1434] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785: Time spent starting tablet: real 0.021s	user 0.006s	sys 0.013s
02:07:03.803 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 6 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283, tablet=5637f0c18ff54bdd81b35dab09fada4d, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:07:03.804 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.21.14.65:38711
02:07:03.804 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

02:07:03.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:03.804769 21561 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu pid 21842
02:07:03.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:03.822608 22456 heartbeater.cc:344] Connected to a master server at 127.21.14.124:36603
02:07:03.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:03.824244 22461 heartbeater.cc:344] Connected to a master server at 127.21.14.125:33787
02:07:03.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:03.824505 22457 heartbeater.cc:344] Connected to a master server at 127.21.14.126:34291
02:07:03.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:03.825007 22456 heartbeater.cc:461] Registering TS with master...
02:07:03.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:03.825068 22461 heartbeater.cc:461] Registering TS with master...
02:07:03.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:03.826103 22456 heartbeater.cc:507] Master 127.21.14.124:36603 requested a full tablet report, sending...
02:07:03.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:03.826128 22461 heartbeater.cc:507] Master 127.21.14.125:33787 requested a full tablet report, sending...
02:07:03.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:03.825003 22457 heartbeater.cc:461] Registering TS with master...
02:07:03.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:03.827019 22457 heartbeater.cc:507] Master 127.21.14.126:34291 requested a full tablet report, sending...
02:07:03.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:03.829201 21744 ts_manager.cc:194] Re-registered known tserver with Master: 1a735d9d94bb4c2e8d28cad5e18d6785 (127.21.14.67:42423)
02:07:03.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:03.829821 21603 ts_manager.cc:194] Re-registered known tserver with Master: 1a735d9d94bb4c2e8d28cad5e18d6785 (127.21.14.67:42423)
02:07:03.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:03.830999 21964 generic_service.cc:196] Checking for leaks (request via RPC)
02:07:03.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:03.831269 21603 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.21.14.67:34907
02:07:03.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:03.832366 21673 ts_manager.cc:194] Re-registered known tserver with Master: 1a735d9d94bb4c2e8d28cad5e18d6785 (127.21.14.67:42423)
02:07:03.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:03.834136 22457 heartbeater.cc:499] Master 127.21.14.126:34291 was elected leader, sending a full tablet report...
02:07:03.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:03.929317 21561 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu with pid 21842
02:07:03.944 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:07:03.945 [DEBUG - kudu-nio-1] (Connection.java:239) [peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711)] handling channelInactive
02:07:03.945 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711)] cleaning up while in state READY due to: connection closed
02:07:03.956 [DEBUG - pool-2-thread-1] (Connection.java:748) [peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711)] connecting to peer
02:07:03.957 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711
02:07:03.957 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711)] cleaning up while in state CONNECTING due to: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711
02:07:03.957 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 790137564e884f989bbd5c36475181af(127.21.14.65:38711) for tablet 5637f0c18ff54bdd81b35dab09fada4d: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711
02:07:03.979 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:03.979 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423),34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979),790137564e884f989bbd5c36475181af(127.21.14.65:38711)[L]]}]
02:07:03.979 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5637f0c18ff54bdd81b35dab09fada4d, attempt=3, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=34), Traces: [0ms] sending RPC to server 790137564e884f989bbd5c36475181af, [0ms] delaying RPC due to: Illegal state: connection is terminated, [11ms] sending RPC to server 790137564e884f989bbd5c36475181af, [12ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [12ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [33ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=Deferred@54163579(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007fe67c338998@5f995214 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@30b7b47f -> wakeup thread Time-limited test)) after lookup
02:07:03.980 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711)] connecting to peer
02:07:03.981 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711
02:07:03.981 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711)] cleaning up while in state CONNECTING due to: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711
02:07:03.981 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 790137564e884f989bbd5c36475181af(127.21.14.65:38711) for tablet 5637f0c18ff54bdd81b35dab09fada4d: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711
02:07:04.018 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:04.019 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423),34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979),790137564e884f989bbd5c36475181af(127.21.14.65:38711)[L]]}]
02:07:04.019 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5637f0c18ff54bdd81b35dab09fada4d, attempt=5, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=74), Traces: [0ms] sending RPC to server 790137564e884f989bbd5c36475181af, [0ms] delaying RPC due to: Illegal state: connection is terminated, [11ms] sending RPC to server 790137564e884f989bbd5c36475181af, [12ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [12ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [33ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [34ms] sending RPC to server 790137564e884f989bbd5c36475181af, [36ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [36ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [70ms] refreshing cache from master, [70ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [73ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=Deferred@54163579(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007fe67c338998@5f995214 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2135664883 after retry RPC@1926426409), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@30b7b47f -> wakeup thread Time-limited test -> (continuation of Deferred@2135664883 after retry RPC@1926426409))) after lookup
02:07:04.020 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711)] connecting to peer
02:07:04.021 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711
02:07:04.021 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711)] cleaning up while in state CONNECTING due to: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711
02:07:04.021 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 790137564e884f989bbd5c36475181af(127.21.14.65:38711) for tablet 5637f0c18ff54bdd81b35dab09fada4d: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711
02:07:04.098 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:04.098 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423),34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979),790137564e884f989bbd5c36475181af(127.21.14.65:38711)[L]]}]
02:07:04.099 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5637f0c18ff54bdd81b35dab09fada4d, attempt=7, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=154), Traces: [0ms] sending RPC to server 790137564e884f989bbd5c36475181af, [0ms] delaying RPC due to: Illegal state: connection is terminated, [11ms] sending RPC to server 790137564e884f989bbd5c36475181af, [12ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [12ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [33ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [34ms] sending RPC to server 790137564e884f989bbd5c36475181af, [36ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [36ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [70ms] refreshing cache from master, [70ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [73ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [75ms] sending RPC to server 790137564e884f989bbd5c36475181af, [76ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [76ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [150ms] refreshing cache from master, [150ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [153ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=Deferred@54163579(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007fe67c338998@5f995214 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2135664883 after retry RPC@1926426409) -> (continuation of Deferred@1245406404 after retry RPC@796225805), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@30b7b47f -> wakeup thread Time-limited test -> (continuation of Deferred@2135664883 after retry RPC@1926426409) -> (continuation of Deferred@1245406404 after retry RPC@796225805))) after lookup
02:07:04.100 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711)] connecting to peer
02:07:04.101 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711
02:07:04.101 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711)] cleaning up while in state CONNECTING due to: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711
02:07:04.101 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 790137564e884f989bbd5c36475181af(127.21.14.65:38711) for tablet 5637f0c18ff54bdd81b35dab09fada4d: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711
02:07:04.139 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:04.139 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423),34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979),790137564e884f989bbd5c36475181af(127.21.14.65:38711)[L]]}]
02:07:04.139 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5637f0c18ff54bdd81b35dab09fada4d, attempt=9, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=194), Traces: [0ms] sending RPC to server 790137564e884f989bbd5c36475181af, [0ms] delaying RPC due to: Illegal state: connection is terminated, [11ms] sending RPC to server 790137564e884f989bbd5c36475181af, [12ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [12ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [33ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [34ms] sending RPC to server 790137564e884f989bbd5c36475181af, [36ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [36ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [70ms] refreshing cache from master, [70ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [73ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [75ms] sending RPC to server 790137564e884f989bbd5c36475181af, [76ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [76ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [150ms] refreshing cache from master, [150ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [153ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [154ms] sending RPC to server 790137564e884f989bbd5c36475181af, [156ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [156ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [191ms] refreshing cache from master, [191ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [193ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=Deferred@54163579(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007fe67c338998@5f995214 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2135664883 after retry RPC@1926426409) -> (continuation of Deferred@1245406404 after retry RPC@796225805) -> (continuation of Deferred@1180131173 after retry RPC@1110356923), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@30b7b47f -> wakeup thread Time-limited test -> (continuation of Deferred@2135664883 after retry RPC@1926426409) -> (continuation of Deferred@1245406404 after retry RPC@796225805) -> (continuation of Deferred@1180131173 after retry RPC@1110356923))) after lookup
02:07:04.140 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711)] connecting to peer
02:07:04.141 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711
02:07:04.141 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711)] cleaning up while in state CONNECTING due to: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711
02:07:04.141 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 790137564e884f989bbd5c36475181af(127.21.14.65:38711) for tablet 5637f0c18ff54bdd81b35dab09fada4d: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711
02:07:04.318 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:04.318 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423),34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979),790137564e884f989bbd5c36475181af(127.21.14.65:38711)[L]]}]
02:07:04.319 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5637f0c18ff54bdd81b35dab09fada4d, attempt=11, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=374), Traces: [0ms] sending RPC to server 790137564e884f989bbd5c36475181af, [0ms] delaying RPC due to: Illegal state: connection is terminated, [11ms] sending RPC to server 790137564e884f989bbd5c36475181af, [12ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [12ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [33ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [34ms] sending RPC to server 790137564e884f989bbd5c36475181af, [36ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [36ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [70ms] refreshing cache from master, [70ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [73ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [75ms] sending RPC to server 790137564e884f989bbd5c36475181af, [76ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [76ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [150ms] refreshing cache from master, [150ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [153ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [154ms] sending RPC to server 790137564e884f989bbd5c36475181af, [156ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [156ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [191ms] refreshing cache from master, [191ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [193ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [195ms] sending RPC to server 790137564e884f989bbd5c36475181af, [196ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [196ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [370ms] refreshing cache from master, [370ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [373ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=Deferred@54163579(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007fe67c338998@5f995214 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2135664883 after retry RPC@1926426409) -> (continuation of Deferred@1245406404 after retry RPC@796225805) -> (continuation of Deferred@1180131173 after retry RPC@1110356923) -> (continuation of Deferred@36650612 after retry RPC@892069601), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@30b7b47f -> wakeup thread Time-limited test -> (continuation of Deferred@2135664883 after retry RPC@1926426409) -> (continuation of Deferred@1245406404 after retry RPC@796225805) -> (continuation of Deferred@1180131173 after retry RPC@1110356923) -> (continuation of Deferred@36650612 after retry RPC@892069601))) after lookup
02:07:04.320 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711)] connecting to peer
02:07:04.322 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711
02:07:04.322 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711)] cleaning up while in state CONNECTING due to: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711
02:07:04.322 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 790137564e884f989bbd5c36475181af(127.21.14.65:38711) for tablet 5637f0c18ff54bdd81b35dab09fada4d: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711
02:07:04.539 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:04.539 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423),34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979),790137564e884f989bbd5c36475181af(127.21.14.65:38711)[L]]}]
02:07:04.540 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5637f0c18ff54bdd81b35dab09fada4d, attempt=13, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=594), Traces: [0ms] sending RPC to server 790137564e884f989bbd5c36475181af, [0ms] delaying RPC due to: Illegal state: connection is terminated, [11ms] sending RPC to server 790137564e884f989bbd5c36475181af, [12ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [12ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [33ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [34ms] sending RPC to server 790137564e884f989bbd5c36475181af, [36ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [36ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [70ms] refreshing cache from master, [70ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [73ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [75ms] sending RPC to server 790137564e884f989bbd5c36475181af, [76ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [76ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [150ms] refreshing cache from master, [150ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [153ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [154ms] sending RPC to server 790137564e884f989bbd5c36475181af, [156ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [156ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [191ms] refreshing cache from master, [191ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [193ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [195ms] sending RPC to server 790137564e884f989bbd5c36475181af, [196ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [196ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [370ms] refreshing cache from master, [370ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [373ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [375ms] sending RPC to server 790137564e884f989bbd5c36475181af, [377ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [377ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [591ms] refreshing cache from master, [591ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [594ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=Deferred@54163579(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007fe67c338998@5f995214 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2135664883 after retry RPC@1926426409) -> (continuation of Deferred@1245406404 after retry RPC@796225805) -> (continuation of Deferred@1180131173 after retry RPC@1110356923) -> (continuation of Deferred@36650612 after retry RPC@892069601) -> (continuation of Deferred@147060410 after retry RPC@1342358074), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@30b7b47f -> wakeup thread Time-limited test -> (continuation of Deferred@2135664883 after retry RPC@1926426409) -> (continuation of Deferred@1245406404 after retry RPC@796225805) -> (continuation of Deferred@1180131173 after retry RPC@1110356923) -> (continuation of Deferred@36650612 after retry RPC@892069601) -> (continuation of Deferred@147060410 after retry RPC@1342358074))) after lookup
02:07:04.541 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711)] connecting to peer
02:07:04.542 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711
02:07:04.542 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711)] cleaning up while in state CONNECTING due to: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711
02:07:04.542 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 790137564e884f989bbd5c36475181af(127.21.14.65:38711) for tablet 5637f0c18ff54bdd81b35dab09fada4d: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711
02:07:05.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:05.132418 22470 raft_consensus.cc:493] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
02:07:05.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:05.132692 22470 raft_consensus.cc:515] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:05.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:05.134264 22470 leader_election.cc:290] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 34e2861a29784992b9d1a9a5876087a6 (127.21.14.66:35979), 790137564e884f989bbd5c36475181af (127.21.14.65:38711)
02:07:05.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:05.136085 22343 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.14.65:38711: connect: Connection refused (error 111)
02:07:05.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:05.137544 22343 leader_election.cc:336] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 790137564e884f989bbd5c36475181af (127.21.14.65:38711): Network error: Client connection negotiation failed: client connection to 127.21.14.65:38711: connect: Connection refused (error 111)
02:07:05.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:05.140798 22061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5637f0c18ff54bdd81b35dab09fada4d" candidate_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "34e2861a29784992b9d1a9a5876087a6" is_pre_election: true
02:07:05.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:05.141520 22342 leader_election.cc:304] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1a735d9d94bb4c2e8d28cad5e18d6785; no voters: 34e2861a29784992b9d1a9a5876087a6, 790137564e884f989bbd5c36475181af
02:07:05.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:05.141858 22470 raft_consensus.cc:2749] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
02:07:05.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:05.430112 22475 raft_consensus.cc:493] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 790137564e884f989bbd5c36475181af)
02:07:05.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:05.430289 22475 raft_consensus.cc:515] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:05.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:05.430908 22475 leader_election.cc:290] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1a735d9d94bb4c2e8d28cad5e18d6785 (127.21.14.67:42423), 790137564e884f989bbd5c36475181af (127.21.14.65:38711)
02:07:05.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:05.433588 21995 leader_election.cc:336] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 790137564e884f989bbd5c36475181af (127.21.14.65:38711): Network error: Client connection negotiation failed: client connection to 127.21.14.65:38711: connect: Connection refused (error 111)
02:07:05.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:05.438258 22410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5637f0c18ff54bdd81b35dab09fada4d" candidate_uuid: "34e2861a29784992b9d1a9a5876087a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" is_pre_election: true
02:07:05.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:05.438584 22410 raft_consensus.cc:2468] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34e2861a29784992b9d1a9a5876087a6 in term 1.
02:07:05.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:05.439370 21995 leader_election.cc:304] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1a735d9d94bb4c2e8d28cad5e18d6785, 34e2861a29784992b9d1a9a5876087a6; no voters: 790137564e884f989bbd5c36475181af
02:07:05.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:05.439641 22475 raft_consensus.cc:2804] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [term 2 FOLLOWER]: Leader pre-election won for term 3
02:07:05.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:05.439735 22475 raft_consensus.cc:493] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 790137564e884f989bbd5c36475181af)
02:07:05.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:05.439798 22475 raft_consensus.cc:3060] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [term 2 FOLLOWER]: Advancing to term 3
02:07:05.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:05.441788 22475 raft_consensus.cc:515] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:05.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:05.442317 22475 leader_election.cc:290] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [CANDIDATE]: Term 3 election: Requested vote from peers 1a735d9d94bb4c2e8d28cad5e18d6785 (127.21.14.67:42423), 790137564e884f989bbd5c36475181af (127.21.14.65:38711)
02:07:05.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:05.442765 22410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5637f0c18ff54bdd81b35dab09fada4d" candidate_uuid: "34e2861a29784992b9d1a9a5876087a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785"
02:07:05.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:05.442952 22410 raft_consensus.cc:3060] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [term 1 FOLLOWER]: Advancing to term 3
02:07:05.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:05.443728 21995 leader_election.cc:336] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 790137564e884f989bbd5c36475181af (127.21.14.65:38711): Network error: Client connection negotiation failed: client connection to 127.21.14.65:38711: connect: Connection refused (error 111)
02:07:05.446 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:05.446622 22410 raft_consensus.cc:2468] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34e2861a29784992b9d1a9a5876087a6 in term 3.
02:07:05.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:05.447013 21995 leader_election.cc:304] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1a735d9d94bb4c2e8d28cad5e18d6785, 34e2861a29784992b9d1a9a5876087a6; no voters: 790137564e884f989bbd5c36475181af
02:07:05.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:05.447261 22475 raft_consensus.cc:2804] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [term 3 FOLLOWER]: Leader election won for term 3
02:07:05.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:05.447606 22475 raft_consensus.cc:697] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [term 3 LEADER]: Becoming Leader. State: Replica: 34e2861a29784992b9d1a9a5876087a6, State: Running, Role: LEADER
02:07:05.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:05.448045 22475 consensus_queue.cc:237] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:05.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:05.451357 21603 catalog_manager.cc:5697] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 reported cstate change: term changed from 2 to 3, leader changed from 790137564e884f989bbd5c36475181af (127.21.14.65) to 34e2861a29784992b9d1a9a5876087a6 (127.21.14.66). New cstate: current_term: 3 leader_uuid: "34e2861a29784992b9d1a9a5876087a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } health_report { overall_health: UNKNOWN } } }
02:07:05.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:05.859963 22410 raft_consensus.cc:1275] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [term 3 FOLLOWER]: Refusing update from remote peer 34e2861a29784992b9d1a9a5876087a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
02:07:05.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:05.860724 22475 consensus_queue.cc:1048] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 4, Time since last communication: 0.000s
02:07:05.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:05.874233 21995 consensus_peers.cc:597] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 -> Peer 790137564e884f989bbd5c36475181af (127.21.14.65:38711): Couldn't send request to peer 790137564e884f989bbd5c36475181af. Status: Network error: Client connection negotiation failed: client connection to 127.21.14.65:38711: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
02:07:06.239 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:06.240 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423),34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)[L],790137564e884f989bbd5c36475181af(127.21.14.65:38711)]}]
02:07:06.240 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5637f0c18ff54bdd81b35dab09fada4d, attempt=15, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=2295), Traces: [0ms] sending RPC to server 790137564e884f989bbd5c36475181af, [0ms] delaying RPC due to: Illegal state: connection is terminated, [11ms] sending RPC to server 790137564e884f989bbd5c36475181af, [12ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [12ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [33ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [34ms] sending RPC to server 790137564e884f989bbd5c36475181af, [36ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [36ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [70ms] refreshing cache from master, [70ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [73ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [75ms] sending RPC to server 790137564e884f989bbd5c36475181af, [76ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [76ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [150ms] refreshing cache from master, [150ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [153ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [154ms] sending RPC to server 790137564e884f989bbd5c36475181af, [156ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [156ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [191ms] refreshing cache from master, [191ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [193ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [195ms] sending RPC to server 790137564e884f989bbd5c36475181af, [196ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [196ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [370ms] refreshing cache from master, [370ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [373ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [375ms] sending RPC to server 790137564e884f989bbd5c36475181af, [377ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [377ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [591ms] refreshing cache from master, [591ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [594ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [595ms] sending RPC to server 790137564e884f989bbd5c36475181af, [597ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [597ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [2291ms] refreshing cache from master, [2291ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [2294ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=Deferred@54163579(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007fe67c338998@5f995214 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2135664883 after retry RPC@1926426409) -> (continuation of Deferred@1245406404 after retry RPC@796225805) -> (continuation of Deferred@1180131173 after retry RPC@1110356923) -> (continuation of Deferred@36650612 after retry RPC@892069601) -> (continuation of Deferred@147060410 after retry RPC@1342358074) -> (continuation of Deferred@1144603026 after retry RPC@493839378), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@30b7b47f -> wakeup thread Time-limited test -> (continuation of Deferred@2135664883 after retry RPC@1926426409) -> (continuation of Deferred@1245406404 after retry RPC@796225805) -> (continuation of Deferred@1180131173 after retry RPC@1110356923) -> (continuation of Deferred@36650612 after retry RPC@892069601) -> (continuation of Deferred@147060410 after retry RPC@1342358074) -> (continuation of Deferred@1144603026 after retry RPC@493839378))) after lookup
02:07:06.241 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)] connecting to peer
02:07:06.242 [DEBUG - kudu-nio-1] (Connection.java:757) [peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)] Successfully connected to peer
02:07:06.257 [DEBUG - kudu-nio-1] (Negotiator.java:918) Authenticated connection [id: 0xfec79e52, L:/127.0.0.1:40838 - R:/127.21.14.66:35979] using TOKEN/null
02:07:06.286 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.21.14.65:38711
02:07:06.287 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

02:07:06.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:06.287856 21561 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu
02:07:06.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu
02:07:06.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-0/wal
02:07:06.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-0/data
02:07:06.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:07:06.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:07:06.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:07:06.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:07:06.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:07:06.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:07:06.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-0/logs
02:07:06.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-0/data/info.pb
02:07:06.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:07:06.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:07:06.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:07:06.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:07:06.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:07:06.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:07:06.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:07:06.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:07:06.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:07:06.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:07:06.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.14.65:38711
02:07:06.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.21.14.65
02:07:06.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.21.14.126:34291,127.21.14.125:33787,127.21.14.124:36603
02:07:06.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=36395
02:07:06.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.14.65
02:07:06.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.14.84:44189
02:07:06.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:07:06.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:07:06.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:07:06.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:06.596889 22504 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:07:06.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:06.597144 22504 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:07:06.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:06.597223 22504 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:07:06.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:06.605173 22504 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:07:06.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:06.605315 22504 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.14.65
02:07:06.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:06.614809 22504 tablet_server_runner.cc:78] Tablet server non-default flags:
02:07:06.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:07:06.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.14.84:44189
02:07:06.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:07:06.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:07:06.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-0/data
02:07:06.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-0/wal
02:07:06.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:07:06.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:07:06.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.14.65:38711
02:07:06.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:07:06.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:07:06.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:07:06.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-0/data/info.pb
02:07:06.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.14.65
02:07:06.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=36395
02:07:06.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.21.14.126:34291,127.21.14.125:33787,127.21.14.124:36603
02:07:06.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:07:06.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:07:06.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:07:06.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:07:06.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:07:06.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.21.14.65
02:07:06.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-0/logs
02:07:06.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:07:06.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:07:06.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:07:06.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:07:06.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:07:06.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
02:07:06.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:07:06.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:43:30 UTC on bdcb31816ec0
02:07:06.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12766
02:07:06.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
02:07:06.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:06.616252 22504 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:07:06.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:06.617991 22504 file_cache.cc:504] Constructed file cache file cache with capacity 419430
02:07:06.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:06.626780 22511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:06.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:06.627398 22514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:06.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:06.627748 22512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:06.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:06.628368 22504 server_base.cc:1061] running on GCE node
02:07:06.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:06.628901 22504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:07:06.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:06.629722 22504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:07:06.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:06.630941 22504 hybrid_clock.cc:648] HybridClock initialized: now 1783476426630888 us; error 40 us; skew 500 ppm
02:07:06.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:06.631222 22504 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:07:06.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:06.633214 22504 webserver.cc:533] Webserver started at http://127.21.14.65:36395/ using document root <none> and password file <none>
02:07:06.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:06.634001 22504 fs_manager.cc:362] Metadata directory not provided
02:07:06.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:06.634130 22504 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:07:06.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:06.638873 22504 fs_manager.cc:714] Time spent opening directory manager: real 0.003s	user 0.002s	sys 0.002s
02:07:06.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:06.641387 22520 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:07:06.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:06.642508 22504 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
02:07:06.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:06.642625 22504 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-0/data,/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-0/wal
02:07:06.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "790137564e884f989bbd5c36475181af"
02:07:06.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:06:59 on dist-test-slave-545r"
02:07:06.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:06.643260 22504 fs_report.cc:389] FS layout report
02:07:06.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:07:06.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-0/wal
02:07:06.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-0/wal
02:07:06.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-0/data/data
02:07:06.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:07:06.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:07:06.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:07:06.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:07:06.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:07:06.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:07:06.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:07:06.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:07:06.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:07:06.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:07:06.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:07:06.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:06.665453 22504 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:07:06.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:06.666167 22504 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:07:06.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:06.666381 22504 kserver.cc:163] Server-wide thread pool size limit: 3276
02:07:06.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:06.667225 22504 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:07:06.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:06.669240 22527 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
02:07:06.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:06.674703 22504 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
02:07:06.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:06.674793 22504 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.006s	user 0.001s	sys 0.000s
02:07:06.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:06.674883 22504 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
02:07:06.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:06.677852 22504 ts_tablet_manager.cc:616] Registered 1 tablets
02:07:06.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:06.677956 22504 ts_tablet_manager.cc:595] Time spent register tablets: real 0.003s	user 0.002s	sys 0.000s
02:07:06.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:06.678421 22527 tablet_bootstrap.cc:492] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af: Bootstrap starting.
02:07:06.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:06.702750 22527 log.cc:826] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af: Log is configured to *not* fsync() on all Append() calls
02:07:06.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:06.712885 22504 rpc_server.cc:307] RPC server started. Bound to: 127.21.14.65:38711
02:07:06.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:06.712922 22634 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.14.65:38711 every 8 connection(s)
02:07:06.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:06.714450 22504 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-0/data/info.pb
02:07:06.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:06.724185 22527 tablet_bootstrap.cc:492] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=6 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
02:07:06.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:06.724545 21561 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu as pid 22504
02:07:06.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:06.724836 22527 tablet_bootstrap.cc:492] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af: Bootstrap complete.
02:07:06.725 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:07:06.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:06.726234 22527 ts_tablet_manager.cc:1403] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af: Time spent bootstrapping tablet: real 0.048s	user 0.031s	sys 0.012s
02:07:06.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:06.732191 22527 raft_consensus.cc:359] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:06.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:06.733084 22527 raft_consensus.cc:740] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 790137564e884f989bbd5c36475181af, State: Initialized, Role: FOLLOWER
02:07:06.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:06.733704 22527 consensus_queue.cc:260] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:06.736 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 9 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283, tablet=5637f0c18ff54bdd81b35dab09fada4d, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:07:06.737 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.21.14.66:35979
02:07:06.738 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

02:07:06.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:06.738425 21561 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu pid 21979
02:07:06.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:06.741506 22527 ts_tablet_manager.cc:1434] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af: Time spent starting tablet: real 0.015s	user 0.006s	sys 0.009s
02:07:06.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:06.762358 22101 generic_service.cc:196] Checking for leaks (request via RPC)
02:07:06.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:06.766288 22635 heartbeater.cc:344] Connected to a master server at 127.21.14.124:36603
02:07:06.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:06.766593 22635 heartbeater.cc:461] Registering TS with master...
02:07:06.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:06.767625 22635 heartbeater.cc:507] Master 127.21.14.124:36603 requested a full tablet report, sending...
02:07:06.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:06.770563 21744 ts_manager.cc:194] Re-registered known tserver with Master: 790137564e884f989bbd5c36475181af (127.21.14.65:38711)
02:07:06.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:06.772951 22637 heartbeater.cc:344] Connected to a master server at 127.21.14.125:33787
02:07:06.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:06.773089 22637 heartbeater.cc:461] Registering TS with master...
02:07:06.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:06.773550 22637 heartbeater.cc:507] Master 127.21.14.125:33787 requested a full tablet report, sending...
02:07:06.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:06.774163 22636 heartbeater.cc:344] Connected to a master server at 127.21.14.126:34291
02:07:06.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:06.774271 22636 heartbeater.cc:461] Registering TS with master...
02:07:06.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:06.774695 22636 heartbeater.cc:507] Master 127.21.14.126:34291 requested a full tablet report, sending...
02:07:06.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:06.774932 21673 ts_manager.cc:194] Re-registered known tserver with Master: 790137564e884f989bbd5c36475181af (127.21.14.65:38711)
02:07:06.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:06.776512 21603 ts_manager.cc:194] Re-registered known tserver with Master: 790137564e884f989bbd5c36475181af (127.21.14.65:38711)
02:07:06.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:06.777913 21603 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.21.14.65:54621
02:07:06.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:06.780189 22636 heartbeater.cc:499] Master 127.21.14.126:34291 was elected leader, sending a full tablet report...
02:07:06.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:06.870810 21561 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu with pid 21979
02:07:06.886 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:07:06.886 [DEBUG - kudu-nio-1] (Connection.java:239) [peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)] handling channelInactive
02:07:06.887 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)] cleaning up while in state READY due to: connection closed
02:07:06.896 [DEBUG - pool-2-thread-1] (Connection.java:748) [peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)] connecting to peer
02:07:06.897 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979
02:07:06.897 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)] cleaning up while in state CONNECTING due to: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979
02:07:06.898 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979) for tablet 5637f0c18ff54bdd81b35dab09fada4d: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979
02:07:06.919 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:06.919 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423),34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)[L],790137564e884f989bbd5c36475181af(127.21.14.65:38711)]}]
02:07:06.919 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5637f0c18ff54bdd81b35dab09fada4d, attempt=3, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=32), Traces: [0ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [1ms] delaying RPC due to: Illegal state: connection is terminated, [10ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [12ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [12ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [30ms] refreshing cache from master, [30ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [32ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=Deferred@2105289272(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007fe67c338998@1bd58e71 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@24f9f596 -> wakeup thread Time-limited test)) after lookup
02:07:06.920 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)] connecting to peer
02:07:06.921 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979
02:07:06.921 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)] cleaning up while in state CONNECTING due to: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979
02:07:06.921 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979) for tablet 5637f0c18ff54bdd81b35dab09fada4d: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979
02:07:06.959 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:06.959 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423),34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)[L],790137564e884f989bbd5c36475181af(127.21.14.65:38711)]}]
02:07:06.960 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5637f0c18ff54bdd81b35dab09fada4d, attempt=5, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=73), Traces: [0ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [1ms] delaying RPC due to: Illegal state: connection is terminated, [10ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [12ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [12ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [30ms] refreshing cache from master, [30ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [32ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [33ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [35ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [35ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [70ms] refreshing cache from master, [70ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [72ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=Deferred@2105289272(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007fe67c338998@1bd58e71 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1112771313 after retry RPC@703431272), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@24f9f596 -> wakeup thread Time-limited test -> (continuation of Deferred@1112771313 after retry RPC@703431272))) after lookup
02:07:06.960 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)] connecting to peer
02:07:06.961 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979
02:07:06.961 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)] cleaning up while in state CONNECTING due to: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979
02:07:06.962 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979) for tablet 5637f0c18ff54bdd81b35dab09fada4d: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979
02:07:07.018 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:07.019 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423),34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)[L],790137564e884f989bbd5c36475181af(127.21.14.65:38711)]}]
02:07:07.020 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5637f0c18ff54bdd81b35dab09fada4d, attempt=7, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=133), Traces: [0ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [1ms] delaying RPC due to: Illegal state: connection is terminated, [10ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [12ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [12ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [30ms] refreshing cache from master, [30ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [32ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [33ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [35ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [35ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [70ms] refreshing cache from master, [70ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [72ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [74ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [76ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [76ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [129ms] refreshing cache from master, [130ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [132ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=Deferred@2105289272(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007fe67c338998@1bd58e71 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1112771313 after retry RPC@703431272) -> (continuation of Deferred@1698939376 after retry RPC@553181682), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@24f9f596 -> wakeup thread Time-limited test -> (continuation of Deferred@1112771313 after retry RPC@703431272) -> (continuation of Deferred@1698939376 after retry RPC@553181682))) after lookup
02:07:07.021 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)] connecting to peer
02:07:07.022 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979
02:07:07.022 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)] cleaning up while in state CONNECTING due to: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979
02:07:07.022 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979) for tablet 5637f0c18ff54bdd81b35dab09fada4d: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979
02:07:07.038 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:07.039 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423),34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)[L],790137564e884f989bbd5c36475181af(127.21.14.65:38711)]}]
02:07:07.039 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5637f0c18ff54bdd81b35dab09fada4d, attempt=9, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=152), Traces: [0ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [1ms] delaying RPC due to: Illegal state: connection is terminated, [10ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [12ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [12ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [30ms] refreshing cache from master, [30ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [32ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [33ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [35ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [35ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [70ms] refreshing cache from master, [70ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [72ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [74ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [76ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [76ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [129ms] refreshing cache from master, [130ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [132ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [134ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [136ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [136ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [150ms] refreshing cache from master, [150ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [152ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=Deferred@2105289272(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007fe67c338998@1bd58e71 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1112771313 after retry RPC@703431272) -> (continuation of Deferred@1698939376 after retry RPC@553181682) -> (continuation of Deferred@108097948 after retry RPC@1642198993), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@24f9f596 -> wakeup thread Time-limited test -> (continuation of Deferred@1112771313 after retry RPC@703431272) -> (continuation of Deferred@1698939376 after retry RPC@553181682) -> (continuation of Deferred@108097948 after retry RPC@1642198993))) after lookup
02:07:07.040 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)] connecting to peer
02:07:07.041 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979
02:07:07.041 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)] cleaning up while in state CONNECTING due to: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979
02:07:07.041 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979) for tablet 5637f0c18ff54bdd81b35dab09fada4d: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979
02:07:07.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:07.784621 22648 raft_consensus.cc:493] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 34e2861a29784992b9d1a9a5876087a6)
02:07:07.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:07.784808 22648 raft_consensus.cc:515] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:07.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:07.785488 22648 leader_election.cc:290] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 34e2861a29784992b9d1a9a5876087a6 (127.21.14.66:35979), 790137564e884f989bbd5c36475181af (127.21.14.65:38711)
02:07:07.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:07.787304 22342 leader_election.cc:336] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 34e2861a29784992b9d1a9a5876087a6 (127.21.14.66:35979): Network error: Client connection negotiation failed: client connection to 127.21.14.66:35979: connect: Connection refused (error 111)
02:07:07.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:07.792531 22589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5637f0c18ff54bdd81b35dab09fada4d" candidate_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "790137564e884f989bbd5c36475181af" is_pre_election: true
02:07:07.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:07.792927 22589 raft_consensus.cc:2468] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a735d9d94bb4c2e8d28cad5e18d6785 in term 2.
02:07:07.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:07.793730 22343 leader_election.cc:304] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1a735d9d94bb4c2e8d28cad5e18d6785, 790137564e884f989bbd5c36475181af; no voters: 34e2861a29784992b9d1a9a5876087a6
02:07:07.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:07.793993 22648 raft_consensus.cc:2804] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [term 3 FOLLOWER]: Leader pre-election won for term 4
02:07:07.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:07.794111 22648 raft_consensus.cc:493] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 34e2861a29784992b9d1a9a5876087a6)
02:07:07.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:07.794210 22648 raft_consensus.cc:3060] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [term 3 FOLLOWER]: Advancing to term 4
02:07:07.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:07.796204 22648 raft_consensus.cc:515] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:07.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:07.796780 22648 leader_election.cc:290] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [CANDIDATE]: Term 4 election: Requested vote from peers 34e2861a29784992b9d1a9a5876087a6 (127.21.14.66:35979), 790137564e884f989bbd5c36475181af (127.21.14.65:38711)
02:07:07.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:07.797304 22589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5637f0c18ff54bdd81b35dab09fada4d" candidate_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "790137564e884f989bbd5c36475181af"
02:07:07.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:07.797472 22589 raft_consensus.cc:3060] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 2 FOLLOWER]: Advancing to term 4
02:07:07.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:07.798053 22342 leader_election.cc:336] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 34e2861a29784992b9d1a9a5876087a6 (127.21.14.66:35979): Network error: Client connection negotiation failed: client connection to 127.21.14.66:35979: connect: Connection refused (error 111)
02:07:07.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:07.801375 22589 raft_consensus.cc:2468] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a735d9d94bb4c2e8d28cad5e18d6785 in term 4.
02:07:07.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:07.801726 22343 leader_election.cc:304] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1a735d9d94bb4c2e8d28cad5e18d6785, 790137564e884f989bbd5c36475181af; no voters: 34e2861a29784992b9d1a9a5876087a6
02:07:07.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:07.801993 22648 raft_consensus.cc:2804] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [term 4 FOLLOWER]: Leader election won for term 4
02:07:07.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:07.802394 22648 raft_consensus.cc:697] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [term 4 LEADER]: Becoming Leader. State: Replica: 1a735d9d94bb4c2e8d28cad5e18d6785, State: Running, Role: LEADER
02:07:07.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:07.802872 22648 consensus_queue.cc:237] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 12, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:07.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:07.806038 21603 catalog_manager.cc:5697] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 reported cstate change: term changed from 3 to 4, leader changed from 34e2861a29784992b9d1a9a5876087a6 (127.21.14.66) to 1a735d9d94bb4c2e8d28cad5e18d6785 (127.21.14.67). New cstate: current_term: 4 leader_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } health_report { overall_health: UNKNOWN } } }
02:07:07.838 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:07.838 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)[L],34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979),790137564e884f989bbd5c36475181af(127.21.14.65:38711)]}]
02:07:07.839 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5637f0c18ff54bdd81b35dab09fada4d, attempt=11, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=952), Traces: [0ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [1ms] delaying RPC due to: Illegal state: connection is terminated, [10ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [12ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [12ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [30ms] refreshing cache from master, [30ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [32ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [33ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [35ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [35ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [70ms] refreshing cache from master, [70ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [72ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [74ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [76ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [76ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [129ms] refreshing cache from master, [130ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [132ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [134ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [136ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [136ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [150ms] refreshing cache from master, [150ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [152ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [154ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [155ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [155ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [949ms] refreshing cache from master, [949ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [952ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=Deferred@2105289272(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007fe67c338998@1bd58e71 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1112771313 after retry RPC@703431272) -> (continuation of Deferred@1698939376 after retry RPC@553181682) -> (continuation of Deferred@108097948 after retry RPC@1642198993) -> (continuation of Deferred@1672236900 after retry RPC@228422591), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@24f9f596 -> wakeup thread Time-limited test -> (continuation of Deferred@1112771313 after retry RPC@703431272) -> (continuation of Deferred@1698939376 after retry RPC@553181682) -> (continuation of Deferred@108097948 after retry RPC@1642198993) -> (continuation of Deferred@1672236900 after retry RPC@228422591))) after lookup
02:07:07.839 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)] connecting to peer
02:07:07.840 [DEBUG - kudu-nio-7] (Connection.java:757) [peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)] Successfully connected to peer
02:07:07.870 [DEBUG - kudu-nio-7] (Negotiator.java:918) Authenticated connection [id: 0xa10322b2, L:/127.0.0.1:49324 - R:/127.21.14.67:42423] using TOKEN/null
02:07:07.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:07.875978 22589 raft_consensus.cc:1275] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 4 FOLLOWER]: Refusing update from remote peer 1a735d9d94bb4c2e8d28cad5e18d6785: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 4 index: 14. (index mismatch)
02:07:07.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:07.876863 22342 consensus_peers.cc:597] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 -> Peer 34e2861a29784992b9d1a9a5876087a6 (127.21.14.66:35979): Couldn't send request to peer 34e2861a29784992b9d1a9a5876087a6. Status: Network error: Client connection negotiation failed: client connection to 127.21.14.66:35979: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
02:07:07.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:07.876871 22648 consensus_queue.cc:1048] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [LEADER]: Connected to new peer: Peer: permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 8, Time since last communication: 0.000s
02:07:07.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:07.890937 22668 mvcc.cc:204] Tried to move back new op lower bound from 7305119448570757120 to 7305119448283811840. Current Snapshot: MvccSnapshot[applied={T|T < 7305119448570757120}]
02:07:07.911 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.21.14.66:35979
02:07:07.911 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

02:07:07.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:07.912170 21561 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu
02:07:07.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu
02:07:07.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-1/wal
02:07:07.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-1/data
02:07:07.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:07:07.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:07:07.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:07:07.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:07:07.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:07:07.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:07:07.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-1/logs
02:07:07.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-1/data/info.pb
02:07:07.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:07:07.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:07:07.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:07:07.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:07:07.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:07:07.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:07:07.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:07:07.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:07:07.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:07:07.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:07:07.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.14.66:35979
02:07:07.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.21.14.66
02:07:07.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.21.14.126:34291,127.21.14.125:33787,127.21.14.124:36603
02:07:07.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=43569
02:07:07.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.14.66
02:07:07.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.14.84:44189
02:07:07.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:07:07.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:07:07.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:07:08.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:08.217862 22676 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:07:08.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:08.218129 22676 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:07:08.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:08.218204 22676 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:07:08.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:08.226274 22676 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:07:08.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:08.226414 22676 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.14.66
02:07:08.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:08.235803 22676 tablet_server_runner.cc:78] Tablet server non-default flags:
02:07:08.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:07:08.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.14.84:44189
02:07:08.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:07:08.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:07:08.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-1/data
02:07:08.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-1/wal
02:07:08.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:07:08.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:07:08.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.14.66:35979
02:07:08.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:07:08.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:07:08.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:07:08.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-1/data/info.pb
02:07:08.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.14.66
02:07:08.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=43569
02:07:08.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.21.14.126:34291,127.21.14.125:33787,127.21.14.124:36603
02:07:08.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:07:08.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:07:08.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:07:08.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:07:08.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:07:08.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.21.14.66
02:07:08.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-1/logs
02:07:08.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:07:08.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:07:08.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:07:08.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:07:08.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:07:08.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
02:07:08.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:07:08.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:43:30 UTC on bdcb31816ec0
02:07:08.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12766
02:07:08.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
02:07:08.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:08.237732 22676 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:07:08.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:08.239974 22676 file_cache.cc:504] Constructed file cache file cache with capacity 419430
02:07:08.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:08.249706 22682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:08.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:08.249683 22681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:08.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:08.250555 22684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:08.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:08.250993 22676 server_base.cc:1061] running on GCE node
02:07:08.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:08.251525 22676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:07:08.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:08.252386 22676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:07:08.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:08.253626 22676 hybrid_clock.cc:648] HybridClock initialized: now 1783476428253584 us; error 29 us; skew 500 ppm
02:07:08.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:08.253943 22676 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:07:08.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:08.255751 22676 webserver.cc:533] Webserver started at http://127.21.14.66:43569/ using document root <none> and password file <none>
02:07:08.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:08.256371 22676 fs_manager.cc:362] Metadata directory not provided
02:07:08.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:08.256503 22676 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:07:08.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:08.261238 22676 fs_manager.cc:714] Time spent opening directory manager: real 0.003s	user 0.001s	sys 0.003s
02:07:08.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:08.263720 22690 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:07:08.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:08.264976 22676 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.001s
02:07:08.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:08.265100 22676 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-1/data,/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-1/wal
02:07:08.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "34e2861a29784992b9d1a9a5876087a6"
02:07:08.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:06:59 on dist-test-slave-545r"
02:07:08.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:08.265821 22676 fs_report.cc:389] FS layout report
02:07:08.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:07:08.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-1/wal
02:07:08.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-1/wal
02:07:08.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-1/data/data
02:07:08.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:07:08.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:07:08.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:07:08.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:07:08.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:07:08.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:07:08.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:07:08.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:07:08.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:07:08.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:07:08.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:07:08.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:08.278718 22676 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:07:08.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:08.279395 22676 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:07:08.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:08.279615 22676 kserver.cc:163] Server-wide thread pool size limit: 3276
02:07:08.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:08.280385 22676 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:07:08.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:08.282289 22697 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
02:07:08.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:08.287775 22676 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
02:07:08.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:08.287849 22676 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.006s	user 0.001s	sys 0.000s
02:07:08.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:08.287937 22676 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
02:07:08.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:08.290767 22676 ts_tablet_manager.cc:616] Registered 1 tablets
02:07:08.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:08.290846 22676 ts_tablet_manager.cc:595] Time spent register tablets: real 0.003s	user 0.001s	sys 0.000s
02:07:08.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:08.291256 22697 tablet_bootstrap.cc:492] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6: Bootstrap starting.
02:07:08.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:08.310685 22697 log.cc:826] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6: Log is configured to *not* fsync() on all Append() calls
02:07:08.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:08.332230 22697 tablet_bootstrap.cc:492] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=9 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
02:07:08.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:08.332852 22697 tablet_bootstrap.cc:492] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6: Bootstrap complete.
02:07:08.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:08.333729 22697 ts_tablet_manager.cc:1403] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6: Time spent bootstrapping tablet: real 0.043s	user 0.018s	sys 0.023s
02:07:08.341 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:08.340003 22697 raft_consensus.cc:359] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:08.341 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:08.341082 22697 raft_consensus.cc:740] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34e2861a29784992b9d1a9a5876087a6, State: Initialized, Role: FOLLOWER
02:07:08.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:08.341904 22697 consensus_queue.cc:260] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 3.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:08.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:08.343012 22676 rpc_server.cc:307] RPC server started. Bound to: 127.21.14.66:35979
02:07:08.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:08.343060 22805 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.14.66:35979 every 8 connection(s)
02:07:08.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:08.343714 22697 ts_tablet_manager.cc:1434] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6: Time spent starting tablet: real 0.010s	user 0.009s	sys 0.001s
02:07:08.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:08.344230 22676 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-1/data/info.pb
02:07:08.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:08.345329 21561 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu as pid 22676
02:07:08.345 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:07:08.371 [DEBUG - kudu-nio-7] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 12 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283, tablet=5637f0c18ff54bdd81b35dab09fada4d, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:07:08.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:08.375510 22807 heartbeater.cc:344] Connected to a master server at 127.21.14.126:34291
02:07:08.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:08.375957 22807 heartbeater.cc:461] Registering TS with master...
02:07:08.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:08.376799 22806 heartbeater.cc:344] Connected to a master server at 127.21.14.124:36603
02:07:08.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:08.376936 22806 heartbeater.cc:461] Registering TS with master...
02:07:08.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:08.377453 22807 heartbeater.cc:507] Master 127.21.14.126:34291 requested a full tablet report, sending...
02:07:08.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:08.378125 22806 heartbeater.cc:507] Master 127.21.14.124:36603 requested a full tablet report, sending...
02:07:08.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:08.379968 22808 heartbeater.cc:344] Connected to a master server at 127.21.14.125:33787
02:07:08.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:08.380124 22808 heartbeater.cc:461] Registering TS with master...
02:07:08.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:08.380309 21603 ts_manager.cc:194] Re-registered known tserver with Master: 34e2861a29784992b9d1a9a5876087a6 (127.21.14.66:35979)
02:07:08.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:08.380558 22808 heartbeater.cc:507] Master 127.21.14.125:33787 requested a full tablet report, sending...
02:07:08.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:08.381783 21603 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.21.14.66:40089
02:07:08.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:08.381914 21673 ts_manager.cc:194] Re-registered known tserver with Master: 34e2861a29784992b9d1a9a5876087a6 (127.21.14.66:35979)
02:07:08.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:08.382128 21744 ts_manager.cc:194] Re-registered known tserver with Master: 34e2861a29784992b9d1a9a5876087a6 (127.21.14.66:35979)
02:07:08.384 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.21.14.67:42423
02:07:08.384 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

02:07:08.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:08.384894 21561 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu pid 22327
02:07:08.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:08.399509 22451 generic_service.cc:196] Checking for leaks (request via RPC)
02:07:08.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:08.497197 21561 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu with pid 22327
02:07:08.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:08.510833 22522 connection.cc:573] server connection from 127.21.14.67:52367 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
02:07:08.511 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:07:08.511 [DEBUG - kudu-nio-7] (Connection.java:239) [peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)] handling channelInactive
02:07:08.511 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)] cleaning up while in state READY due to: connection closed
02:07:08.512 [DEBUG - Time-limited test] (Connection.java:748) [peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)] connecting to peer
02:07:08.513 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:08.513 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:08.513 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423) for tablet 5637f0c18ff54bdd81b35dab09fada4d: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:08.518 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:08.519 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)[L],34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979),790137564e884f989bbd5c36475181af(127.21.14.65:38711)]}]
02:07:08.519 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5637f0c18ff54bdd81b35dab09fada4d, attempt=2, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=7), Traces: [0ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [1ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [1ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [6ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=Deferred@865251773(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007fe67c338998@3bb02910 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7d3e1ff -> wakeup thread Time-limited test)) after lookup
02:07:08.520 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)] connecting to peer
02:07:08.521 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:08.521 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:08.521 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423) for tablet 5637f0c18ff54bdd81b35dab09fada4d: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:08.538 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:08.538 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)[L],34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979),790137564e884f989bbd5c36475181af(127.21.14.65:38711)]}]
02:07:08.538 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5637f0c18ff54bdd81b35dab09fada4d, attempt=4, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=26), Traces: [0ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [1ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [1ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [6ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [8ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [9ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [9ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [23ms] refreshing cache from master, [23ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [26ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=Deferred@865251773(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007fe67c338998@3bb02910 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1975836622 after retry RPC@1837110416), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7d3e1ff -> wakeup thread Time-limited test -> (continuation of Deferred@1975836622 after retry RPC@1837110416))) after lookup
02:07:08.539 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)] connecting to peer
02:07:08.539 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:08.539 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:08.540 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423) for tablet 5637f0c18ff54bdd81b35dab09fada4d: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:08.578 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:08.578 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)[L],34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979),790137564e884f989bbd5c36475181af(127.21.14.65:38711)]}]
02:07:08.579 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5637f0c18ff54bdd81b35dab09fada4d, attempt=6, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=67), Traces: [0ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [1ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [1ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [6ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [8ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [9ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [9ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [23ms] refreshing cache from master, [23ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [26ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [26ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [28ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [28ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [66ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=Deferred@865251773(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007fe67c338998@3bb02910 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1975836622 after retry RPC@1837110416) -> (continuation of Deferred@1390556985 after retry RPC@1059822945), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7d3e1ff -> wakeup thread Time-limited test -> (continuation of Deferred@1975836622 after retry RPC@1837110416) -> (continuation of Deferred@1390556985 after retry RPC@1059822945))) after lookup
02:07:08.579 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)] connecting to peer
02:07:08.581 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:08.581 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:08.581 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423) for tablet 5637f0c18ff54bdd81b35dab09fada4d: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:08.679 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:08.679 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)[L],34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979),790137564e884f989bbd5c36475181af(127.21.14.65:38711)]}]
02:07:08.679 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5637f0c18ff54bdd81b35dab09fada4d, attempt=8, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=167), Traces: [0ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [1ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [1ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [6ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [8ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [9ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [9ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [23ms] refreshing cache from master, [23ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [26ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [26ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [28ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [28ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [66ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [67ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [69ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [69ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [164ms] refreshing cache from master, [164ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [166ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=Deferred@865251773(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007fe67c338998@3bb02910 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1975836622 after retry RPC@1837110416) -> (continuation of Deferred@1390556985 after retry RPC@1059822945) -> (continuation of Deferred@257302969 after retry RPC@1828021671), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7d3e1ff -> wakeup thread Time-limited test -> (continuation of Deferred@1975836622 after retry RPC@1837110416) -> (continuation of Deferred@1390556985 after retry RPC@1059822945) -> (continuation of Deferred@257302969 after retry RPC@1828021671))) after lookup
02:07:08.680 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)] connecting to peer
02:07:08.681 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:08.681 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:08.681 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423) for tablet 5637f0c18ff54bdd81b35dab09fada4d: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:08.859 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:08.859 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)[L],34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979),790137564e884f989bbd5c36475181af(127.21.14.65:38711)]}]
02:07:08.860 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5637f0c18ff54bdd81b35dab09fada4d, attempt=10, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=348), Traces: [0ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [1ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [1ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [6ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [8ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [9ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [9ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [23ms] refreshing cache from master, [23ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [26ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [26ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [28ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [28ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [66ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [67ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [69ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [69ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [164ms] refreshing cache from master, [164ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [166ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [168ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [169ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [169ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [344ms] refreshing cache from master, [344ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [347ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=Deferred@865251773(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007fe67c338998@3bb02910 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1975836622 after retry RPC@1837110416) -> (continuation of Deferred@1390556985 after retry RPC@1059822945) -> (continuation of Deferred@257302969 after retry RPC@1828021671) -> (continuation of Deferred@690744713 after retry RPC@1857318230), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7d3e1ff -> wakeup thread Time-limited test -> (continuation of Deferred@1975836622 after retry RPC@1837110416) -> (continuation of Deferred@1390556985 after retry RPC@1059822945) -> (continuation of Deferred@257302969 after retry RPC@1828021671) -> (continuation of Deferred@690744713 after retry RPC@1857318230))) after lookup
02:07:08.861 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)] connecting to peer
02:07:08.862 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:08.862 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:08.862 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423) for tablet 5637f0c18ff54bdd81b35dab09fada4d: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:09.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:09.385593 22807 heartbeater.cc:499] Master 127.21.14.126:34291 was elected leader, sending a full tablet report...
02:07:09.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:09.792779 22820 raft_consensus.cc:493] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
02:07:09.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:09.793051 22820 raft_consensus.cc:515] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:09.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:09.794730 22820 leader_election.cc:290] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1a735d9d94bb4c2e8d28cad5e18d6785 (127.21.14.67:42423), 790137564e884f989bbd5c36475181af (127.21.14.65:38711)
02:07:09.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:09.796419 22692 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.14.67:42423: connect: Connection refused (error 111)
02:07:09.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:09.797971 22692 leader_election.cc:336] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1a735d9d94bb4c2e8d28cad5e18d6785 (127.21.14.67:42423): Network error: Client connection negotiation failed: client connection to 127.21.14.67:42423: connect: Connection refused (error 111)
02:07:09.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:09.801793 22589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5637f0c18ff54bdd81b35dab09fada4d" candidate_uuid: "34e2861a29784992b9d1a9a5876087a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "790137564e884f989bbd5c36475181af" is_pre_election: true
02:07:09.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:09.802528 22692 leader_election.cc:304] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 34e2861a29784992b9d1a9a5876087a6; no voters: 1a735d9d94bb4c2e8d28cad5e18d6785, 790137564e884f989bbd5c36475181af
02:07:09.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:09.802838 22820 raft_consensus.cc:2749] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
02:07:09.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:09.998903 22825 raft_consensus.cc:493] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1a735d9d94bb4c2e8d28cad5e18d6785)
02:07:09.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:09.999123 22825 raft_consensus.cc:515] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:10.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:10.000586 22825 leader_election.cc:290] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1a735d9d94bb4c2e8d28cad5e18d6785 (127.21.14.67:42423), 34e2861a29784992b9d1a9a5876087a6 (127.21.14.66:35979)
02:07:10.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:10.001680 22522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.14.67:42423: connect: Connection refused (error 111)
02:07:10.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:10.003057 22522 leader_election.cc:336] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 1a735d9d94bb4c2e8d28cad5e18d6785 (127.21.14.67:42423): Network error: Client connection negotiation failed: client connection to 127.21.14.67:42423: connect: Connection refused (error 111)
02:07:10.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:10.008790 22760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5637f0c18ff54bdd81b35dab09fada4d" candidate_uuid: "790137564e884f989bbd5c36475181af" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "34e2861a29784992b9d1a9a5876087a6" is_pre_election: true
02:07:10.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:10.009132 22760 raft_consensus.cc:2468] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 790137564e884f989bbd5c36475181af in term 3.
02:07:10.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:10.009931 22521 leader_election.cc:304] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 34e2861a29784992b9d1a9a5876087a6, 790137564e884f989bbd5c36475181af; no voters: 1a735d9d94bb4c2e8d28cad5e18d6785
02:07:10.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:10.010250 22825 raft_consensus.cc:2804] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 4 FOLLOWER]: Leader pre-election won for term 5
02:07:10.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:10.010366 22825 raft_consensus.cc:493] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1a735d9d94bb4c2e8d28cad5e18d6785)
02:07:10.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:10.010459 22825 raft_consensus.cc:3060] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 4 FOLLOWER]: Advancing to term 5
02:07:10.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:10.012365 22825 raft_consensus.cc:515] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:10.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:10.012920 22825 leader_election.cc:290] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [CANDIDATE]: Term 5 election: Requested vote from peers 1a735d9d94bb4c2e8d28cad5e18d6785 (127.21.14.67:42423), 34e2861a29784992b9d1a9a5876087a6 (127.21.14.66:35979)
02:07:10.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:10.013417 22760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5637f0c18ff54bdd81b35dab09fada4d" candidate_uuid: "790137564e884f989bbd5c36475181af" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "34e2861a29784992b9d1a9a5876087a6"
02:07:10.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:10.013608 22760 raft_consensus.cc:3060] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [term 3 FOLLOWER]: Advancing to term 5
02:07:10.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:10.014158 22522 leader_election.cc:336] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 1a735d9d94bb4c2e8d28cad5e18d6785 (127.21.14.67:42423): Network error: Client connection negotiation failed: client connection to 127.21.14.67:42423: connect: Connection refused (error 111)
02:07:10.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:10.017270 22760 raft_consensus.cc:2468] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 790137564e884f989bbd5c36475181af in term 5.
02:07:10.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:10.017620 22521 leader_election.cc:304] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 34e2861a29784992b9d1a9a5876087a6, 790137564e884f989bbd5c36475181af; no voters: 1a735d9d94bb4c2e8d28cad5e18d6785
02:07:10.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:10.017889 22825 raft_consensus.cc:2804] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 5 FOLLOWER]: Leader election won for term 5
02:07:10.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:10.018298 22825 raft_consensus.cc:697] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 5 LEADER]: Becoming Leader. State: Replica: 790137564e884f989bbd5c36475181af, State: Running, Role: LEADER
02:07:10.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:10.018819 22825 consensus_queue.cc:237] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 4.16, Last appended by leader: 16, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:10.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:10.022192 21603 catalog_manager.cc:5697] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af reported cstate change: term changed from 4 to 5, leader changed from 1a735d9d94bb4c2e8d28cad5e18d6785 (127.21.14.67) to 790137564e884f989bbd5c36475181af (127.21.14.65). New cstate: current_term: 5 leader_uuid: "790137564e884f989bbd5c36475181af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } health_report { overall_health: HEALTHY } } }
02:07:10.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:10.583892 22522 consensus_peers.cc:597] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af -> Peer 1a735d9d94bb4c2e8d28cad5e18d6785 (127.21.14.67:42423): Couldn't send request to peer 1a735d9d94bb4c2e8d28cad5e18d6785. Status: Network error: Client connection negotiation failed: client connection to 127.21.14.67:42423: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
02:07:10.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:10.619831 22760 raft_consensus.cc:1275] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [term 5 FOLLOWER]: Refusing update from remote peer 790137564e884f989bbd5c36475181af: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 5 index: 17. (index mismatch)
02:07:10.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:10.620726 22842 consensus_queue.cc:1048] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [LEADER]: Connected to new peer: Peer: permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 12, Time since last communication: 0.000s
02:07:10.919 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:10.919 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423),34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979),790137564e884f989bbd5c36475181af(127.21.14.65:38711)[L]]}]
02:07:10.919 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5637f0c18ff54bdd81b35dab09fada4d, attempt=12, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=2407), Traces: [0ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [1ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [1ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [6ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [8ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [9ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [9ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [23ms] refreshing cache from master, [23ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [26ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [26ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [28ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [28ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [66ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [67ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [69ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [69ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [164ms] refreshing cache from master, [164ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [166ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [168ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [169ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [169ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [344ms] refreshing cache from master, [344ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [347ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [348ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [350ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [350ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [2403ms] refreshing cache from master, [2404ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [2406ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=Deferred@865251773(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007fe67c338998@3bb02910 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1975836622 after retry RPC@1837110416) -> (continuation of Deferred@1390556985 after retry RPC@1059822945) -> (continuation of Deferred@257302969 after retry RPC@1828021671) -> (continuation of Deferred@690744713 after retry RPC@1857318230) -> (continuation of Deferred@673284781 after retry RPC@1697813538), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7d3e1ff -> wakeup thread Time-limited test -> (continuation of Deferred@1975836622 after retry RPC@1837110416) -> (continuation of Deferred@1390556985 after retry RPC@1059822945) -> (continuation of Deferred@257302969 after retry RPC@1828021671) -> (continuation of Deferred@690744713 after retry RPC@1857318230) -> (continuation of Deferred@673284781 after retry RPC@1697813538))) after lookup
02:07:10.920 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711)] connecting to peer
02:07:10.921 [DEBUG - kudu-nio-6] (Connection.java:757) [peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711)] Successfully connected to peer
02:07:10.949 [DEBUG - kudu-nio-6] (Negotiator.java:918) Authenticated connection [id: 0x1f7233d3, L:/127.0.0.1:53412 - R:/127.21.14.65:38711] using TOKEN/null
02:07:10.978 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.21.14.67:42423
02:07:10.979 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

02:07:10.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:10.979777 21561 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu
02:07:10.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu
02:07:10.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/wal
02:07:10.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/data
02:07:10.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:07:10.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:07:10.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:07:10.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:07:10.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:07:10.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:07:10.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/logs
02:07:10.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/data/info.pb
02:07:10.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:07:10.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:07:10.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:07:10.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:07:10.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:07:10.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:07:10.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:07:10.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:07:10.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:07:10.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:07:10.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.14.67:42423
02:07:10.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.21.14.67
02:07:10.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.21.14.126:34291,127.21.14.125:33787,127.21.14.124:36603
02:07:10.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=37049
02:07:10.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.14.67
02:07:10.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.14.84:44189
02:07:10.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:07:10.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:07:10.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:07:11.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:11.289038 22856 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:07:11.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:11.289304 22856 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:07:11.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:11.289384 22856 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:07:11.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:11.297394 22856 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:07:11.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:11.297534 22856 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.14.67
02:07:11.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:11.306854 22856 tablet_server_runner.cc:78] Tablet server non-default flags:
02:07:11.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:07:11.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.14.84:44189
02:07:11.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:07:11.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:07:11.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/data
02:07:11.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/wal
02:07:11.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:07:11.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:07:11.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.14.67:42423
02:07:11.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:07:11.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:07:11.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:07:11.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/data/info.pb
02:07:11.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.14.67
02:07:11.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=37049
02:07:11.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.21.14.126:34291,127.21.14.125:33787,127.21.14.124:36603
02:07:11.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:07:11.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:07:11.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:07:11.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:07:11.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:07:11.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.21.14.67
02:07:11.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/logs
02:07:11.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:07:11.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:07:11.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:07:11.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:07:11.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:07:11.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
02:07:11.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:07:11.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:43:30 UTC on bdcb31816ec0
02:07:11.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12766
02:07:11.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
02:07:11.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:11.308501 22856 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:07:11.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:11.310706 22856 file_cache.cc:504] Constructed file cache file cache with capacity 419430
02:07:11.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:11.319514 22864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:11.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:11.319548 22863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:11.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:11.320045 22866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:11.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:11.320966 22856 server_base.cc:1061] running on GCE node
02:07:11.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:11.321458 22856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:07:11.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:11.322252 22856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:07:11.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:11.323433 22856 hybrid_clock.cc:648] HybridClock initialized: now 1783476431323377 us; error 47 us; skew 500 ppm
02:07:11.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:11.323679 22856 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:07:11.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:11.325480 22856 webserver.cc:533] Webserver started at http://127.21.14.67:37049/ using document root <none> and password file <none>
02:07:11.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:11.326098 22856 fs_manager.cc:362] Metadata directory not provided
02:07:11.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:11.326184 22856 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:07:11.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:11.330804 22856 fs_manager.cc:714] Time spent opening directory manager: real 0.003s	user 0.004s	sys 0.000s
02:07:11.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:11.333297 22872 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:07:11.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:11.334407 22856 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
02:07:11.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:11.334523 22856 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/data,/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/wal
02:07:11.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "1a735d9d94bb4c2e8d28cad5e18d6785"
02:07:11.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:06:59 on dist-test-slave-545r"
02:07:11.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:11.335124 22856 fs_report.cc:389] FS layout report
02:07:11.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:07:11.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/wal
02:07:11.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/wal
02:07:11.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/data/data
02:07:11.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:07:11.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:07:11.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:07:11.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:07:11.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:07:11.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:07:11.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:07:11.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:07:11.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:07:11.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:07:11.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:07:11.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:11.355710 22856 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:07:11.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:11.356439 22856 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:07:11.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:11.356698 22856 kserver.cc:163] Server-wide thread pool size limit: 3276
02:07:11.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:11.357515 22856 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:07:11.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:11.359375 22879 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
02:07:11.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:11.364820 22856 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
02:07:11.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:11.364893 22856 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.006s	user 0.001s	sys 0.000s
02:07:11.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:11.364979 22856 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
02:07:11.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:11.367740 22856 ts_tablet_manager.cc:616] Registered 1 tablets
02:07:11.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:11.367820 22856 ts_tablet_manager.cc:595] Time spent register tablets: real 0.003s	user 0.000s	sys 0.000s
02:07:11.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:11.368223 22879 tablet_bootstrap.cc:492] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785: Bootstrap starting.
02:07:11.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:11.390116 22879 log.cc:826] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785: Log is configured to *not* fsync() on all Append() calls
02:07:11.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:11.405684 22856 rpc_server.cc:307] RPC server started. Bound to: 127.21.14.67:42423
02:07:11.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:11.405694 22986 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.14.67:42423 every 8 connection(s)
02:07:11.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:11.406957 22856 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/data/info.pb
02:07:11.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:11.413095 21561 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu as pid 22856
02:07:11.413 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:07:11.438 [DEBUG - kudu-nio-6] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 15 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283, tablet=5637f0c18ff54bdd81b35dab09fada4d, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:07:11.438 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.21.14.65:38711
02:07:11.439 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

02:07:11.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:11.439517 21561 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu pid 22504
02:07:11.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:11.440142 22879 tablet_bootstrap.cc:492] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785: Bootstrap replayed 1/1 log segments. Stats: ops{read=16 overwritten=0 applied=16 ignored=0} inserts{seen=12 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
02:07:11.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:11.440840 22879 tablet_bootstrap.cc:492] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785: Bootstrap complete.
02:07:11.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:11.441798 22879 ts_tablet_manager.cc:1403] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785: Time spent bootstrapping tablet: real 0.074s	user 0.029s	sys 0.016s
02:07:11.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:11.447013 22879 raft_consensus.cc:359] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:11.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:11.447896 22879 raft_consensus.cc:740] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a735d9d94bb4c2e8d28cad5e18d6785, State: Initialized, Role: FOLLOWER
02:07:11.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:11.451190 22630 generic_service.cc:196] Checking for leaks (request via RPC)
02:07:11.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:11.451917 22879 consensus_queue.cc:260] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 4.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:11.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:11.452960 22989 heartbeater.cc:344] Connected to a master server at 127.21.14.125:33787
02:07:11.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:11.453315 22989 heartbeater.cc:461] Registering TS with master...
02:07:11.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:11.453872 22987 heartbeater.cc:344] Connected to a master server at 127.21.14.124:36603
02:07:11.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:11.453994 22987 heartbeater.cc:461] Registering TS with master...
02:07:11.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:11.454476 22989 heartbeater.cc:507] Master 127.21.14.125:33787 requested a full tablet report, sending...
02:07:11.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:11.454530 22987 heartbeater.cc:507] Master 127.21.14.124:36603 requested a full tablet report, sending...
02:07:11.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:11.455030 22988 heartbeater.cc:344] Connected to a master server at 127.21.14.126:34291
02:07:11.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:11.455149 22988 heartbeater.cc:461] Registering TS with master...
02:07:11.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:11.455679 22988 heartbeater.cc:507] Master 127.21.14.126:34291 requested a full tablet report, sending...
02:07:11.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:11.457472 21673 ts_manager.cc:194] Re-registered known tserver with Master: 1a735d9d94bb4c2e8d28cad5e18d6785 (127.21.14.67:42423)
02:07:11.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:11.458325 21745 ts_manager.cc:194] Re-registered known tserver with Master: 1a735d9d94bb4c2e8d28cad5e18d6785 (127.21.14.67:42423)
02:07:11.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:11.458444 22879 ts_tablet_manager.cc:1434] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785: Time spent starting tablet: real 0.017s	user 0.009s	sys 0.005s
02:07:11.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:11.459434 21603 ts_manager.cc:194] Re-registered known tserver with Master: 1a735d9d94bb4c2e8d28cad5e18d6785 (127.21.14.67:42423)
02:07:11.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:11.460742 21603 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.21.14.67:34819
02:07:11.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:11.463297 22988 heartbeater.cc:499] Master 127.21.14.126:34291 was elected leader, sending a full tablet report...
02:07:11.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:11.556902 21561 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu with pid 22504
02:07:11.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:11.570214 22693 connection.cc:573] server connection from 127.21.14.65:56615 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
02:07:11.570 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:07:11.571 [DEBUG - kudu-nio-6] (Connection.java:239) [peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711)] handling channelInactive
02:07:11.571 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711)] cleaning up while in state READY due to: connection closed
02:07:11.576 [DEBUG - pool-2-thread-1] (Connection.java:748) [peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711)] connecting to peer
02:07:11.577 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711
02:07:11.577 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711)] cleaning up while in state CONNECTING due to: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711
02:07:11.577 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 790137564e884f989bbd5c36475181af(127.21.14.65:38711) for tablet 5637f0c18ff54bdd81b35dab09fada4d: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711
02:07:11.598 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:11.599 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423),34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979),790137564e884f989bbd5c36475181af(127.21.14.65:38711)[L]]}]
02:07:11.599 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5637f0c18ff54bdd81b35dab09fada4d, attempt=3, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=28), Traces: [0ms] sending RPC to server 790137564e884f989bbd5c36475181af, [0ms] delaying RPC due to: Illegal state: connection is terminated, [5ms] sending RPC to server 790137564e884f989bbd5c36475181af, [6ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [6ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [27ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=Deferred@930756147(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007fe67c338998@4618377a -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@62966f25 -> wakeup thread Time-limited test)) after lookup
02:07:11.599 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711)] connecting to peer
02:07:11.600 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711
02:07:11.600 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711)] cleaning up while in state CONNECTING due to: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711
02:07:11.600 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 790137564e884f989bbd5c36475181af(127.21.14.65:38711) for tablet 5637f0c18ff54bdd81b35dab09fada4d: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711
02:07:11.618 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:11.618 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423),34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979),790137564e884f989bbd5c36475181af(127.21.14.65:38711)[L]]}]
02:07:11.618 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5637f0c18ff54bdd81b35dab09fada4d, attempt=5, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=47), Traces: [0ms] sending RPC to server 790137564e884f989bbd5c36475181af, [0ms] delaying RPC due to: Illegal state: connection is terminated, [5ms] sending RPC to server 790137564e884f989bbd5c36475181af, [6ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [6ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [27ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [28ms] sending RPC to server 790137564e884f989bbd5c36475181af, [29ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [29ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [44ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [46ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=Deferred@930756147(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007fe67c338998@4618377a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1938577675 after retry RPC@460663948), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@62966f25 -> wakeup thread Time-limited test -> (continuation of Deferred@1938577675 after retry RPC@460663948))) after lookup
02:07:11.619 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711)] connecting to peer
02:07:11.619 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711
02:07:11.620 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711)] cleaning up while in state CONNECTING due to: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711
02:07:11.620 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 790137564e884f989bbd5c36475181af(127.21.14.65:38711) for tablet 5637f0c18ff54bdd81b35dab09fada4d: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711
02:07:11.679 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:11.679 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423),34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979),790137564e884f989bbd5c36475181af(127.21.14.65:38711)[L]]}]
02:07:11.679 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5637f0c18ff54bdd81b35dab09fada4d, attempt=7, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=108), Traces: [0ms] sending RPC to server 790137564e884f989bbd5c36475181af, [0ms] delaying RPC due to: Illegal state: connection is terminated, [5ms] sending RPC to server 790137564e884f989bbd5c36475181af, [6ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [6ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [27ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [28ms] sending RPC to server 790137564e884f989bbd5c36475181af, [29ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [29ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [44ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [46ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [47ms] sending RPC to server 790137564e884f989bbd5c36475181af, [49ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [49ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [105ms] refreshing cache from master, [105ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [107ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=Deferred@930756147(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007fe67c338998@4618377a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1938577675 after retry RPC@460663948) -> (continuation of Deferred@784243990 after retry RPC@471713025), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@62966f25 -> wakeup thread Time-limited test -> (continuation of Deferred@1938577675 after retry RPC@460663948) -> (continuation of Deferred@784243990 after retry RPC@471713025))) after lookup
02:07:11.681 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711)] connecting to peer
02:07:11.682 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711
02:07:11.682 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711)] cleaning up while in state CONNECTING due to: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711
02:07:11.682 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 790137564e884f989bbd5c36475181af(127.21.14.65:38711) for tablet 5637f0c18ff54bdd81b35dab09fada4d: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711
02:07:11.898 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:11.898 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423),34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979),790137564e884f989bbd5c36475181af(127.21.14.65:38711)[L]]}]
02:07:11.899 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5637f0c18ff54bdd81b35dab09fada4d, attempt=9, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=327), Traces: [0ms] sending RPC to server 790137564e884f989bbd5c36475181af, [0ms] delaying RPC due to: Illegal state: connection is terminated, [5ms] sending RPC to server 790137564e884f989bbd5c36475181af, [6ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [6ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [27ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [28ms] sending RPC to server 790137564e884f989bbd5c36475181af, [29ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [29ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [44ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [46ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [47ms] sending RPC to server 790137564e884f989bbd5c36475181af, [49ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [49ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [105ms] refreshing cache from master, [105ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [107ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [109ms] sending RPC to server 790137564e884f989bbd5c36475181af, [111ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [111ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [324ms] refreshing cache from master, [325ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [327ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=Deferred@930756147(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007fe67c338998@4618377a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1938577675 after retry RPC@460663948) -> (continuation of Deferred@784243990 after retry RPC@471713025) -> (continuation of Deferred@1041454092 after retry RPC@1280621314), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@62966f25 -> wakeup thread Time-limited test -> (continuation of Deferred@1938577675 after retry RPC@460663948) -> (continuation of Deferred@784243990 after retry RPC@471713025) -> (continuation of Deferred@1041454092 after retry RPC@1280621314))) after lookup
02:07:11.899 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711)] connecting to peer
02:07:11.900 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711
02:07:11.901 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711)] cleaning up while in state CONNECTING due to: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711
02:07:11.901 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 790137564e884f989bbd5c36475181af(127.21.14.65:38711) for tablet 5637f0c18ff54bdd81b35dab09fada4d: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711
02:07:12.358 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:12.358 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423),34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979),790137564e884f989bbd5c36475181af(127.21.14.65:38711)[L]]}]
02:07:12.359 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5637f0c18ff54bdd81b35dab09fada4d, attempt=11, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=787), Traces: [0ms] sending RPC to server 790137564e884f989bbd5c36475181af, [0ms] delaying RPC due to: Illegal state: connection is terminated, [5ms] sending RPC to server 790137564e884f989bbd5c36475181af, [6ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [6ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [27ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [28ms] sending RPC to server 790137564e884f989bbd5c36475181af, [29ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [29ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [44ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [46ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [47ms] sending RPC to server 790137564e884f989bbd5c36475181af, [49ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [49ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [105ms] refreshing cache from master, [105ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [107ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [109ms] sending RPC to server 790137564e884f989bbd5c36475181af, [111ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [111ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [324ms] refreshing cache from master, [325ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [327ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [328ms] sending RPC to server 790137564e884f989bbd5c36475181af, [330ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [330ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [784ms] refreshing cache from master, [784ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [787ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=Deferred@930756147(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007fe67c338998@4618377a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1938577675 after retry RPC@460663948) -> (continuation of Deferred@784243990 after retry RPC@471713025) -> (continuation of Deferred@1041454092 after retry RPC@1280621314) -> (continuation of Deferred@593838227 after retry RPC@1249932118), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@62966f25 -> wakeup thread Time-limited test -> (continuation of Deferred@1938577675 after retry RPC@460663948) -> (continuation of Deferred@784243990 after retry RPC@471713025) -> (continuation of Deferred@1041454092 after retry RPC@1280621314) -> (continuation of Deferred@593838227 after retry RPC@1249932118))) after lookup
02:07:12.359 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711)] connecting to peer
02:07:12.360 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711
02:07:12.360 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711)] cleaning up while in state CONNECTING due to: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711
02:07:12.361 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 790137564e884f989bbd5c36475181af(127.21.14.65:38711) for tablet 5637f0c18ff54bdd81b35dab09fada4d: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711
02:07:12.658 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:12.658 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423),34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979),790137564e884f989bbd5c36475181af(127.21.14.65:38711)[L]]}]
02:07:12.658 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5637f0c18ff54bdd81b35dab09fada4d, attempt=13, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=1087), Traces: [0ms] sending RPC to server 790137564e884f989bbd5c36475181af, [0ms] delaying RPC due to: Illegal state: connection is terminated, [5ms] sending RPC to server 790137564e884f989bbd5c36475181af, [6ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [6ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [27ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [28ms] sending RPC to server 790137564e884f989bbd5c36475181af, [29ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [29ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [44ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [46ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [47ms] sending RPC to server 790137564e884f989bbd5c36475181af, [49ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [49ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [105ms] refreshing cache from master, [105ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [107ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [109ms] sending RPC to server 790137564e884f989bbd5c36475181af, [111ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [111ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [324ms] refreshing cache from master, [325ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [327ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [328ms] sending RPC to server 790137564e884f989bbd5c36475181af, [330ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [330ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [784ms] refreshing cache from master, [784ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [787ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [788ms] sending RPC to server 790137564e884f989bbd5c36475181af, [789ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [790ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [1084ms] refreshing cache from master, [1084ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [1087ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=Deferred@930756147(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007fe67c338998@4618377a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1938577675 after retry RPC@460663948) -> (continuation of Deferred@784243990 after retry RPC@471713025) -> (continuation of Deferred@1041454092 after retry RPC@1280621314) -> (continuation of Deferred@593838227 after retry RPC@1249932118) -> (continuation of Deferred@570526670 after retry RPC@1808946715), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@62966f25 -> wakeup thread Time-limited test -> (continuation of Deferred@1938577675 after retry RPC@460663948) -> (continuation of Deferred@784243990 after retry RPC@471713025) -> (continuation of Deferred@1041454092 after retry RPC@1280621314) -> (continuation of Deferred@593838227 after retry RPC@1249932118) -> (continuation of Deferred@570526670 after retry RPC@1808946715))) after lookup
02:07:12.659 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711)] connecting to peer
02:07:12.660 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711
02:07:12.660 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711)] cleaning up while in state CONNECTING due to: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711
02:07:12.661 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 790137564e884f989bbd5c36475181af(127.21.14.65:38711) for tablet 5637f0c18ff54bdd81b35dab09fada4d: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711
02:07:13.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:13.058835 23001 raft_consensus.cc:493] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 790137564e884f989bbd5c36475181af)
02:07:13.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:13.059005 23001 raft_consensus.cc:515] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:13.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:13.059585 23001 leader_election.cc:290] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1a735d9d94bb4c2e8d28cad5e18d6785 (127.21.14.67:42423), 790137564e884f989bbd5c36475181af (127.21.14.65:38711)
02:07:13.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:13.062108 22692 leader_election.cc:336] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 790137564e884f989bbd5c36475181af (127.21.14.65:38711): Network error: Client connection negotiation failed: client connection to 127.21.14.65:38711: connect: Connection refused (error 111)
02:07:13.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:13.067435 22942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5637f0c18ff54bdd81b35dab09fada4d" candidate_uuid: "34e2861a29784992b9d1a9a5876087a6" candidate_term: 6 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" is_pre_election: true
02:07:13.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:13.067890 22942 raft_consensus.cc:2468] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34e2861a29784992b9d1a9a5876087a6 in term 4.
02:07:13.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:13.068776 22692 leader_election.cc:304] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1a735d9d94bb4c2e8d28cad5e18d6785, 34e2861a29784992b9d1a9a5876087a6; no voters: 790137564e884f989bbd5c36475181af
02:07:13.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:13.069036 23001 raft_consensus.cc:2804] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [term 5 FOLLOWER]: Leader pre-election won for term 6
02:07:13.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:13.069144 23001 raft_consensus.cc:493] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 790137564e884f989bbd5c36475181af)
02:07:13.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:13.069217 23001 raft_consensus.cc:3060] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [term 5 FOLLOWER]: Advancing to term 6
02:07:13.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:13.071733 23001 raft_consensus.cc:515] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:13.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:13.072292 23001 leader_election.cc:290] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [CANDIDATE]: Term 6 election: Requested vote from peers 1a735d9d94bb4c2e8d28cad5e18d6785 (127.21.14.67:42423), 790137564e884f989bbd5c36475181af (127.21.14.65:38711)
02:07:13.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:13.072700 22942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5637f0c18ff54bdd81b35dab09fada4d" candidate_uuid: "34e2861a29784992b9d1a9a5876087a6" candidate_term: 6 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785"
02:07:13.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:13.072857 22942 raft_consensus.cc:3060] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [term 4 FOLLOWER]: Advancing to term 6
02:07:13.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:13.073848 22692 leader_election.cc:336] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 790137564e884f989bbd5c36475181af (127.21.14.65:38711): Network error: Client connection negotiation failed: client connection to 127.21.14.65:38711: connect: Connection refused (error 111)
02:07:13.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:13.076776 22942 raft_consensus.cc:2468] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34e2861a29784992b9d1a9a5876087a6 in term 6.
02:07:13.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:13.077193 22692 leader_election.cc:304] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1a735d9d94bb4c2e8d28cad5e18d6785, 34e2861a29784992b9d1a9a5876087a6; no voters: 790137564e884f989bbd5c36475181af
02:07:13.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:13.077461 23001 raft_consensus.cc:2804] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [term 6 FOLLOWER]: Leader election won for term 6
02:07:13.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:13.077860 23001 raft_consensus.cc:697] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [term 6 LEADER]: Becoming Leader. State: Replica: 34e2861a29784992b9d1a9a5876087a6, State: Running, Role: LEADER
02:07:13.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:13.078331 23001 consensus_queue.cc:237] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 5.20, Last appended by leader: 20, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:13.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:13.081950 21603 catalog_manager.cc:5697] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 reported cstate change: term changed from 5 to 6, leader changed from 790137564e884f989bbd5c36475181af (127.21.14.65) to 34e2861a29784992b9d1a9a5876087a6 (127.21.14.66). New cstate: current_term: 6 leader_uuid: "34e2861a29784992b9d1a9a5876087a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } health_report { overall_health: UNKNOWN } } }
02:07:13.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:13.557447 22692 consensus_peers.cc:597] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 -> Peer 790137564e884f989bbd5c36475181af (127.21.14.65:38711): Couldn't send request to peer 790137564e884f989bbd5c36475181af. Status: Network error: Client connection negotiation failed: client connection to 127.21.14.65:38711: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
02:07:13.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:13.594329 22942 raft_consensus.cc:1275] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [term 6 FOLLOWER]: Refusing update from remote peer 34e2861a29784992b9d1a9a5876087a6: Log matching property violated. Preceding OpId in replica: term: 4 index: 16. Preceding OpId from leader: term: 6 index: 21. (index mismatch)
02:07:13.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:13.595268 23001 consensus_queue.cc:1048] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 16, Time since last communication: 0.000s
02:07:14.760 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:14.760 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423),34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)[L],790137564e884f989bbd5c36475181af(127.21.14.65:38711)]}]
02:07:14.761 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5637f0c18ff54bdd81b35dab09fada4d, attempt=15, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=3190), Traces: [0ms] sending RPC to server 790137564e884f989bbd5c36475181af, [0ms] delaying RPC due to: Illegal state: connection is terminated, [5ms] sending RPC to server 790137564e884f989bbd5c36475181af, [6ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [6ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [27ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [28ms] sending RPC to server 790137564e884f989bbd5c36475181af, [29ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [29ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [44ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [46ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [47ms] sending RPC to server 790137564e884f989bbd5c36475181af, [49ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [49ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [105ms] refreshing cache from master, [105ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [107ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [109ms] sending RPC to server 790137564e884f989bbd5c36475181af, [111ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [111ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [324ms] refreshing cache from master, [325ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [327ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [328ms] sending RPC to server 790137564e884f989bbd5c36475181af, [330ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [330ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [784ms] refreshing cache from master, [784ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [787ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [788ms] sending RPC to server 790137564e884f989bbd5c36475181af, [789ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [790ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [1084ms] refreshing cache from master, [1084ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [1087ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [1088ms] sending RPC to server 790137564e884f989bbd5c36475181af, [1089ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [1090ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [3184ms] refreshing cache from master, [3185ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [3188ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=Deferred@930756147(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007fe67c338998@4618377a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1938577675 after retry RPC@460663948) -> (continuation of Deferred@784243990 after retry RPC@471713025) -> (continuation of Deferred@1041454092 after retry RPC@1280621314) -> (continuation of Deferred@593838227 after retry RPC@1249932118) -> (continuation of Deferred@570526670 after retry RPC@1808946715) -> (continuation of Deferred@488729006 after retry RPC@1681557151), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@62966f25 -> wakeup thread Time-limited test -> (continuation of Deferred@1938577675 after retry RPC@460663948) -> (continuation of Deferred@784243990 after retry RPC@471713025) -> (continuation of Deferred@1041454092 after retry RPC@1280621314) -> (continuation of Deferred@593838227 after retry RPC@1249932118) -> (continuation of Deferred@570526670 after retry RPC@1808946715) -> (continuation of Deferred@488729006 after retry RPC@1681557151))) after lookup
02:07:14.762 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)] connecting to peer
02:07:14.763 [DEBUG - kudu-nio-6] (Connection.java:757) [peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)] Successfully connected to peer
02:07:14.779 [DEBUG - kudu-nio-6] (Negotiator.java:918) Authenticated connection [id: 0x7eb6682f, L:/127.0.0.1:32894 - R:/127.21.14.66:35979] using TOKEN/null
02:07:14.806 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.21.14.65:38711
02:07:14.808 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

02:07:14.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:14.808943 21561 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu
02:07:14.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu
02:07:14.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-0/wal
02:07:14.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-0/data
02:07:14.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:07:14.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:07:14.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:07:14.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:07:14.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:07:14.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:07:14.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-0/logs
02:07:14.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-0/data/info.pb
02:07:14.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:07:14.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:07:14.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:07:14.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:07:14.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:07:14.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:07:14.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:07:14.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:07:14.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:07:14.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:07:14.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.14.65:38711
02:07:14.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.21.14.65
02:07:14.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.21.14.126:34291,127.21.14.125:33787,127.21.14.124:36603
02:07:14.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=36395
02:07:14.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.14.65
02:07:14.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.14.84:44189
02:07:14.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:07:14.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:07:14.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:07:14.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:14.938575 22692 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.14.65:38711: connect: Connection refused (error 111) [suppressed 5 similar messages]
02:07:15.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:15.132452 23039 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:07:15.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:15.132748 23039 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:07:15.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:15.132838 23039 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:07:15.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:15.140986 23039 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:07:15.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:15.141132 23039 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.14.65
02:07:15.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:15.150521 23039 tablet_server_runner.cc:78] Tablet server non-default flags:
02:07:15.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:07:15.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.14.84:44189
02:07:15.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:07:15.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:07:15.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-0/data
02:07:15.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-0/wal
02:07:15.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:07:15.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:07:15.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.14.65:38711
02:07:15.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:07:15.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:07:15.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:07:15.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-0/data/info.pb
02:07:15.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.14.65
02:07:15.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=36395
02:07:15.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.21.14.126:34291,127.21.14.125:33787,127.21.14.124:36603
02:07:15.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:07:15.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:07:15.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:07:15.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:07:15.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:07:15.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.21.14.65
02:07:15.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-0/logs
02:07:15.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:07:15.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:07:15.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:07:15.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:07:15.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:07:15.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
02:07:15.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:07:15.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:43:30 UTC on bdcb31816ec0
02:07:15.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12766
02:07:15.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
02:07:15.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:15.151944 23039 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:07:15.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:15.153600 23039 file_cache.cc:504] Constructed file cache file cache with capacity 419430
02:07:15.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:15.162024 23044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:15.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:15.162055 23045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:15.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:15.162477 23047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:15.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:15.163722 23039 server_base.cc:1061] running on GCE node
02:07:15.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:15.164176 23039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:07:15.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:15.164983 23039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:07:15.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:15.166157 23039 hybrid_clock.cc:648] HybridClock initialized: now 1783476435166093 us; error 56 us; skew 500 ppm
02:07:15.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:15.166401 23039 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:07:15.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:15.168282 23039 webserver.cc:533] Webserver started at http://127.21.14.65:36395/ using document root <none> and password file <none>
02:07:15.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:15.168944 23039 fs_manager.cc:362] Metadata directory not provided
02:07:15.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:15.169025 23039 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:07:15.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:15.173878 23039 fs_manager.cc:714] Time spent opening directory manager: real 0.003s	user 0.004s	sys 0.000s
02:07:15.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:15.176469 23053 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:07:15.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:15.177629 23039 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
02:07:15.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:15.177770 23039 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-0/data,/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-0/wal
02:07:15.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "790137564e884f989bbd5c36475181af"
02:07:15.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:06:59 on dist-test-slave-545r"
02:07:15.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:15.178526 23039 fs_report.cc:389] FS layout report
02:07:15.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:07:15.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-0/wal
02:07:15.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-0/wal
02:07:15.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-0/data/data
02:07:15.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:07:15.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:07:15.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:07:15.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:07:15.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:07:15.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:07:15.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:07:15.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:07:15.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:07:15.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:07:15.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:07:15.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:15.198011 23039 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:07:15.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:15.198764 23039 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:07:15.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:15.199044 23039 kserver.cc:163] Server-wide thread pool size limit: 3276
02:07:15.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:15.199903 23039 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:07:15.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:15.201815 23060 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
02:07:15.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:15.207020 23039 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
02:07:15.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:15.207113 23039 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.006s	user 0.001s	sys 0.000s
02:07:15.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:15.207233 23039 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
02:07:15.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:15.210132 23039 ts_tablet_manager.cc:616] Registered 1 tablets
02:07:15.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:15.210227 23039 ts_tablet_manager.cc:595] Time spent register tablets: real 0.003s	user 0.002s	sys 0.000s
02:07:15.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:15.210564 23060 tablet_bootstrap.cc:492] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af: Bootstrap starting.
02:07:15.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:15.231118 23060 log.cc:826] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af: Log is configured to *not* fsync() on all Append() calls
02:07:15.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:15.252285 23039 rpc_server.cc:307] RPC server started. Bound to: 127.21.14.65:38711
02:07:15.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:15.252331 23167 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.14.65:38711 every 8 connection(s)
02:07:15.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:15.253587 23039 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-0/data/info.pb
02:07:15.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:15.255283 21561 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu as pid 23039
02:07:15.256 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:07:15.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:15.258729 23060 tablet_bootstrap.cc:492] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af: Bootstrap replayed 1/1 log segments. Stats: ops{read=20 overwritten=0 applied=20 ignored=0} inserts{seen=15 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
02:07:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:15.259368 23060 tablet_bootstrap.cc:492] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af: Bootstrap complete.
02:07:15.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:15.261356 23060 ts_tablet_manager.cc:1403] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af: Time spent bootstrapping tablet: real 0.051s	user 0.039s	sys 0.010s
02:07:15.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:15.268544 23060 raft_consensus.cc:359] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:15.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:15.269588 23060 raft_consensus.cc:740] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 790137564e884f989bbd5c36475181af, State: Initialized, Role: FOLLOWER
02:07:15.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:15.270419 23060 consensus_queue.cc:260] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 20, Last appended: 5.20, Last appended by leader: 20, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:15.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:15.273777 23060 ts_tablet_manager.cc:1434] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af: Time spent starting tablet: real 0.012s	user 0.012s	sys 0.000s
02:07:15.283 [DEBUG - kudu-nio-6] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 18 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283, tablet=5637f0c18ff54bdd81b35dab09fada4d, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:07:15.283 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.21.14.66:35979
02:07:15.284 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

02:07:15.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:15.284252 21561 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu pid 22676
02:07:15.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:15.309921 23169 heartbeater.cc:344] Connected to a master server at 127.21.14.126:34291
02:07:15.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:15.310302 23169 heartbeater.cc:461] Registering TS with master...
02:07:15.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:15.311239 23169 heartbeater.cc:507] Master 127.21.14.126:34291 requested a full tablet report, sending...
02:07:15.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:15.313516 23168 heartbeater.cc:344] Connected to a master server at 127.21.14.124:36603
02:07:15.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:15.313679 23168 heartbeater.cc:461] Registering TS with master...
02:07:15.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:15.314127 23168 heartbeater.cc:507] Master 127.21.14.124:36603 requested a full tablet report, sending...
02:07:15.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:15.315008 21601 ts_manager.cc:194] Re-registered known tserver with Master: 790137564e884f989bbd5c36475181af (127.21.14.65:38711)
02:07:15.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:15.316776 21601 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.21.14.65:55145
02:07:15.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:15.318331 21745 ts_manager.cc:194] Re-registered known tserver with Master: 790137564e884f989bbd5c36475181af (127.21.14.65:38711)
02:07:15.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:15.320012 23169 heartbeater.cc:499] Master 127.21.14.126:34291 was elected leader, sending a full tablet report...
02:07:15.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:15.322657 23170 heartbeater.cc:344] Connected to a master server at 127.21.14.125:33787
02:07:15.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:15.322774 23170 heartbeater.cc:461] Registering TS with master...
02:07:15.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:15.322762 22800 generic_service.cc:196] Checking for leaks (request via RPC)
02:07:15.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:15.323051 23170 heartbeater.cc:507] Master 127.21.14.125:33787 requested a full tablet report, sending...
02:07:15.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:15.324819 21673 ts_manager.cc:194] Re-registered known tserver with Master: 790137564e884f989bbd5c36475181af (127.21.14.65:38711)
02:07:15.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:15.434021 21561 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu with pid 22676
02:07:15.448 [DEBUG - kudu-nio-6] (Connection.java:239) [peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)] handling channelInactive
02:07:15.448 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:07:15.448 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)] cleaning up while in state READY due to: connection closed
02:07:15.449 [DEBUG - Time-limited test] (Connection.java:748) [peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)] connecting to peer
02:07:15.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:15.449213 23184 negotiation.cc:336] Failed RPC negotiation. Trace:
02:07:15.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:07:15.428063 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.21.14.66:39197 (local address 127.21.14.65:38711)
02:07:15.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:07:15.436003 (+  7940us) server_negotiation.cc:207] Beginning negotiation
02:07:15.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:07:15.436010 (+     7us) server_negotiation.cc:400] Waiting for connection header
02:07:15.450 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979
02:07:15.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:07:15.436039 (+    29us) server_negotiation.cc:408] Connection header received
02:07:15.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:07:15.436136 (+    97us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
02:07:15.450 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)] cleaning up while in state CONNECTING due to: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979
02:07:15.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:07:15.436142 (+     6us) server_negotiation.cc:462] Received NEGOTIATE request from client
02:07:15.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:07:15.436348 (+   206us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
02:07:15.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:07:15.436548 (+   200us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
02:07:15.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:07:15.447871 (+ 11323us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.14.66:39197: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
02:07:15.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":7802,"thread_start_us":7662,"threads_started":1}
02:07:15.450 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979) for tablet 5637f0c18ff54bdd81b35dab09fada4d: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979
02:07:15.458 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:15.458 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423),34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)[L],790137564e884f989bbd5c36475181af(127.21.14.65:38711)]}]
02:07:15.458 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5637f0c18ff54bdd81b35dab09fada4d, attempt=2, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=9), Traces: [0ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [2ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [2ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [9ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=Deferred@829306776(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007fe67c338998@1d04154 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@635d760c -> wakeup thread Time-limited test)) after lookup
02:07:15.458 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)] connecting to peer
02:07:15.459 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979
02:07:15.459 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)] cleaning up while in state CONNECTING due to: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979
02:07:15.460 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979) for tablet 5637f0c18ff54bdd81b35dab09fada4d: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979
02:07:15.478 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:15.479 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423),34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)[L],790137564e884f989bbd5c36475181af(127.21.14.65:38711)]}]
02:07:15.479 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5637f0c18ff54bdd81b35dab09fada4d, attempt=4, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=30), Traces: [0ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [2ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [2ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [9ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [10ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [11ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [12ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [28ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [30ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=Deferred@829306776(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007fe67c338998@1d04154 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@414159496 after retry RPC@2086301510), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@635d760c -> wakeup thread Time-limited test -> (continuation of Deferred@414159496 after retry RPC@2086301510))) after lookup
02:07:15.479 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)] connecting to peer
02:07:15.480 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979
02:07:15.480 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)] cleaning up while in state CONNECTING due to: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979
02:07:15.480 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979) for tablet 5637f0c18ff54bdd81b35dab09fada4d: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979
02:07:15.497 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:15.497 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423),34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)[L],790137564e884f989bbd5c36475181af(127.21.14.65:38711)]}]
02:07:15.498 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5637f0c18ff54bdd81b35dab09fada4d, attempt=6, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=49), Traces: [0ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [2ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [2ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [9ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [10ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [11ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [12ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [28ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [30ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [31ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [32ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [33ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [47ms] refreshing cache from master, [47ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [49ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=Deferred@829306776(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007fe67c338998@1d04154 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@414159496 after retry RPC@2086301510) -> (continuation of Deferred@1069162384 after retry RPC@280323966), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@635d760c -> wakeup thread Time-limited test -> (continuation of Deferred@414159496 after retry RPC@2086301510) -> (continuation of Deferred@1069162384 after retry RPC@280323966))) after lookup
02:07:15.498 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)] connecting to peer
02:07:15.499 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979
02:07:15.499 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)] cleaning up while in state CONNECTING due to: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979
02:07:15.500 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979) for tablet 5637f0c18ff54bdd81b35dab09fada4d: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979
02:07:15.618 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:15.618 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423),34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)[L],790137564e884f989bbd5c36475181af(127.21.14.65:38711)]}]
02:07:15.619 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5637f0c18ff54bdd81b35dab09fada4d, attempt=8, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=170), Traces: [0ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [2ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [2ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [9ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [10ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [11ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [12ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [28ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [30ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [31ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [32ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [33ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [47ms] refreshing cache from master, [47ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [49ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [50ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [51ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [52ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [167ms] refreshing cache from master, [167ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [170ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=Deferred@829306776(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007fe67c338998@1d04154 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@414159496 after retry RPC@2086301510) -> (continuation of Deferred@1069162384 after retry RPC@280323966) -> (continuation of Deferred@1733814105 after retry RPC@1549988442), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@635d760c -> wakeup thread Time-limited test -> (continuation of Deferred@414159496 after retry RPC@2086301510) -> (continuation of Deferred@1069162384 after retry RPC@280323966) -> (continuation of Deferred@1733814105 after retry RPC@1549988442))) after lookup
02:07:15.620 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)] connecting to peer
02:07:15.621 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979
02:07:15.621 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)] cleaning up while in state CONNECTING due to: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979
02:07:15.621 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979) for tablet 5637f0c18ff54bdd81b35dab09fada4d: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979
02:07:15.839 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:15.839 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423),34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)[L],790137564e884f989bbd5c36475181af(127.21.14.65:38711)]}]
02:07:15.839 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5637f0c18ff54bdd81b35dab09fada4d, attempt=10, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=390), Traces: [0ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [2ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [2ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [9ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [10ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [11ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [12ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [28ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [30ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [31ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [32ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [33ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [47ms] refreshing cache from master, [47ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [49ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [50ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [51ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [52ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [167ms] refreshing cache from master, [167ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [170ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [171ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [173ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [173ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [388ms] refreshing cache from master, [388ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [390ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=Deferred@829306776(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007fe67c338998@1d04154 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@414159496 after retry RPC@2086301510) -> (continuation of Deferred@1069162384 after retry RPC@280323966) -> (continuation of Deferred@1733814105 after retry RPC@1549988442) -> (continuation of Deferred@317590006 after retry RPC@1507183059), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@635d760c -> wakeup thread Time-limited test -> (continuation of Deferred@414159496 after retry RPC@2086301510) -> (continuation of Deferred@1069162384 after retry RPC@280323966) -> (continuation of Deferred@1733814105 after retry RPC@1549988442) -> (continuation of Deferred@317590006 after retry RPC@1507183059))) after lookup
02:07:15.840 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)] connecting to peer
02:07:15.841 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979
02:07:15.841 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)] cleaning up while in state CONNECTING due to: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979
02:07:15.841 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979) for tablet 5637f0c18ff54bdd81b35dab09fada4d: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979
02:07:15.939 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:15.939 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423),34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)[L],790137564e884f989bbd5c36475181af(127.21.14.65:38711)]}]
02:07:15.940 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5637f0c18ff54bdd81b35dab09fada4d, attempt=12, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=491), Traces: [0ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [2ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [2ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [9ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [10ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [11ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [12ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [28ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [30ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [31ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [32ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [33ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [47ms] refreshing cache from master, [47ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [49ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [50ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [51ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [52ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [167ms] refreshing cache from master, [167ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [170ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [171ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [173ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [173ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [388ms] refreshing cache from master, [388ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [390ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [392ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [393ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [393ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [488ms] refreshing cache from master, [488ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [490ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=Deferred@829306776(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007fe67c338998@1d04154 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@414159496 after retry RPC@2086301510) -> (continuation of Deferred@1069162384 after retry RPC@280323966) -> (continuation of Deferred@1733814105 after retry RPC@1549988442) -> (continuation of Deferred@317590006 after retry RPC@1507183059) -> (continuation of Deferred@769075922 after retry RPC@1522263530), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@635d760c -> wakeup thread Time-limited test -> (continuation of Deferred@414159496 after retry RPC@2086301510) -> (continuation of Deferred@1069162384 after retry RPC@280323966) -> (continuation of Deferred@1733814105 after retry RPC@1549988442) -> (continuation of Deferred@317590006 after retry RPC@1507183059) -> (continuation of Deferred@769075922 after retry RPC@1522263530))) after lookup
02:07:15.940 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)] connecting to peer
02:07:15.941 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979
02:07:15.941 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)] cleaning up while in state CONNECTING due to: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979
02:07:15.941 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979) for tablet 5637f0c18ff54bdd81b35dab09fada4d: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979
02:07:16.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:16.632642 23185 raft_consensus.cc:493] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
02:07:16.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:16.632884 23185 raft_consensus.cc:515] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:16.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:16.634469 23185 leader_election.cc:290] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1a735d9d94bb4c2e8d28cad5e18d6785 (127.21.14.67:42423), 34e2861a29784992b9d1a9a5876087a6 (127.21.14.66:35979)
02:07:16.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:16.635995 23054 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.14.66:35979: connect: Connection refused (error 111)
02:07:16.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:16.637554 23054 leader_election.cc:336] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 34e2861a29784992b9d1a9a5876087a6 (127.21.14.66:35979): Network error: Client connection negotiation failed: client connection to 127.21.14.66:35979: connect: Connection refused (error 111)
02:07:16.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:16.640839 22942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5637f0c18ff54bdd81b35dab09fada4d" candidate_uuid: "790137564e884f989bbd5c36475181af" candidate_term: 6 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" is_pre_election: true
02:07:16.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:16.641629 23055 leader_election.cc:304] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 790137564e884f989bbd5c36475181af; no voters: 1a735d9d94bb4c2e8d28cad5e18d6785, 34e2861a29784992b9d1a9a5876087a6
02:07:16.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:16.642011 23185 raft_consensus.cc:2749] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
02:07:16.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:16.929378 23190 raft_consensus.cc:493] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 34e2861a29784992b9d1a9a5876087a6)
02:07:16.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:16.929626 23190 raft_consensus.cc:515] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:16.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:16.931195 23190 leader_election.cc:290] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 34e2861a29784992b9d1a9a5876087a6 (127.21.14.66:35979), 790137564e884f989bbd5c36475181af (127.21.14.65:38711)
02:07:16.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:16.932788 22873 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.14.66:35979: connect: Connection refused (error 111)
02:07:16.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:16.934448 22873 leader_election.cc:336] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 34e2861a29784992b9d1a9a5876087a6 (127.21.14.66:35979): Network error: Client connection negotiation failed: client connection to 127.21.14.66:35979: connect: Connection refused (error 111)
02:07:16.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:16.940310 23123 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5637f0c18ff54bdd81b35dab09fada4d" candidate_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" candidate_term: 7 candidate_status { last_received { term: 6 index: 24 } } ignore_live_leader: false dest_uuid: "790137564e884f989bbd5c36475181af" is_pre_election: true
02:07:16.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:16.940712 23123 raft_consensus.cc:2468] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a735d9d94bb4c2e8d28cad5e18d6785 in term 5.
02:07:16.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:16.941618 22874 leader_election.cc:304] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1a735d9d94bb4c2e8d28cad5e18d6785, 790137564e884f989bbd5c36475181af; no voters: 34e2861a29784992b9d1a9a5876087a6
02:07:16.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:16.942013 23190 raft_consensus.cc:2804] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [term 6 FOLLOWER]: Leader pre-election won for term 7
02:07:16.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:16.942137 23190 raft_consensus.cc:493] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 34e2861a29784992b9d1a9a5876087a6)
02:07:16.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:16.942236 23190 raft_consensus.cc:3060] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [term 6 FOLLOWER]: Advancing to term 7
02:07:16.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:16.944314 23190 raft_consensus.cc:515] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:16.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:16.944926 23190 leader_election.cc:290] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [CANDIDATE]: Term 7 election: Requested vote from peers 34e2861a29784992b9d1a9a5876087a6 (127.21.14.66:35979), 790137564e884f989bbd5c36475181af (127.21.14.65:38711)
02:07:16.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:16.945482 23123 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5637f0c18ff54bdd81b35dab09fada4d" candidate_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" candidate_term: 7 candidate_status { last_received { term: 6 index: 24 } } ignore_live_leader: false dest_uuid: "790137564e884f989bbd5c36475181af"
02:07:16.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:16.945690 23123 raft_consensus.cc:3060] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 5 FOLLOWER]: Advancing to term 7
02:07:16.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:16.946308 22873 leader_election.cc:336] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 34e2861a29784992b9d1a9a5876087a6 (127.21.14.66:35979): Network error: Client connection negotiation failed: client connection to 127.21.14.66:35979: connect: Connection refused (error 111)
02:07:16.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:16.949452 23123 raft_consensus.cc:2468] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a735d9d94bb4c2e8d28cad5e18d6785 in term 7.
02:07:16.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:16.950017 22874 leader_election.cc:304] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1a735d9d94bb4c2e8d28cad5e18d6785, 790137564e884f989bbd5c36475181af; no voters: 34e2861a29784992b9d1a9a5876087a6
02:07:16.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:16.950502 23190 raft_consensus.cc:2804] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [term 7 FOLLOWER]: Leader election won for term 7
02:07:16.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:16.951370 23190 raft_consensus.cc:697] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [term 7 LEADER]: Becoming Leader. State: Replica: 1a735d9d94bb4c2e8d28cad5e18d6785, State: Running, Role: LEADER
02:07:16.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:16.952133 23190 consensus_queue.cc:237] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 6.24, Last appended by leader: 24, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:16.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:16.957144 21601 catalog_manager.cc:5697] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 reported cstate change: term changed from 6 to 7, leader changed from 34e2861a29784992b9d1a9a5876087a6 (127.21.14.66) to 1a735d9d94bb4c2e8d28cad5e18d6785 (127.21.14.67). New cstate: current_term: 7 leader_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } health_report { overall_health: UNKNOWN } } }
02:07:17.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:17.445253 22873 consensus_peers.cc:597] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 -> Peer 34e2861a29784992b9d1a9a5876087a6 (127.21.14.66:35979): Couldn't send request to peer 34e2861a29784992b9d1a9a5876087a6. Status: Network error: Client connection negotiation failed: client connection to 127.21.14.66:35979: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
02:07:17.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:17.541981 23123 raft_consensus.cc:1275] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 7 FOLLOWER]: Refusing update from remote peer 1a735d9d94bb4c2e8d28cad5e18d6785: Log matching property violated. Preceding OpId in replica: term: 5 index: 20. Preceding OpId from leader: term: 7 index: 25. (index mismatch)
02:07:17.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:17.542801 23190 consensus_queue.cc:1048] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [LEADER]: Connected to new peer: Peer: permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 20, Time since last communication: 0.000s
02:07:17.938 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:17.939 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)[L],34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979),790137564e884f989bbd5c36475181af(127.21.14.65:38711)]}]
02:07:17.939 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5637f0c18ff54bdd81b35dab09fada4d, attempt=14, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=2490), Traces: [0ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [2ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [2ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [9ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [10ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [11ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [12ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [28ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [30ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [31ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [32ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [33ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [47ms] refreshing cache from master, [47ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [49ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [50ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [51ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [52ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [167ms] refreshing cache from master, [167ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [170ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [171ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [173ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [173ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [388ms] refreshing cache from master, [388ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [390ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [392ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [393ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [393ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [488ms] refreshing cache from master, [488ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [490ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [492ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [493ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [493ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [2487ms] refreshing cache from master, [2487ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [2490ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=Deferred@829306776(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007fe67c338998@1d04154 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@414159496 after retry RPC@2086301510) -> (continuation of Deferred@1069162384 after retry RPC@280323966) -> (continuation of Deferred@1733814105 after retry RPC@1549988442) -> (continuation of Deferred@317590006 after retry RPC@1507183059) -> (continuation of Deferred@769075922 after retry RPC@1522263530) -> (continuation of Deferred@887646373 after retry RPC@830033767), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@635d760c -> wakeup thread Time-limited test -> (continuation of Deferred@414159496 after retry RPC@2086301510) -> (continuation of Deferred@1069162384 after retry RPC@280323966) -> (continuation of Deferred@1733814105 after retry RPC@1549988442) -> (continuation of Deferred@317590006 after retry RPC@1507183059) -> (continuation of Deferred@769075922 after retry RPC@1522263530) -> (continuation of Deferred@887646373 after retry RPC@830033767))) after lookup
02:07:17.940 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)] connecting to peer
02:07:17.941 [DEBUG - kudu-nio-6] (Connection.java:757) [peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)] Successfully connected to peer
02:07:17.955 [DEBUG - kudu-nio-6] (Negotiator.java:918) Authenticated connection [id: 0xe0ef45f2, L:/127.0.0.1:47820 - R:/127.21.14.67:42423] using TOKEN/null
02:07:17.974 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.21.14.66:35979
02:07:17.975 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

02:07:17.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:17.975620 21561 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu
02:07:17.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu
02:07:17.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-1/wal
02:07:17.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-1/data
02:07:17.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:07:17.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:07:17.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:07:17.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:07:17.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:07:17.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:07:17.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-1/logs
02:07:17.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-1/data/info.pb
02:07:17.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:07:17.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:07:17.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:07:17.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:07:17.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:07:17.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:07:17.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:07:17.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:07:17.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:07:17.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:07:17.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.14.66:35979
02:07:17.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.21.14.66
02:07:17.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.21.14.126:34291,127.21.14.125:33787,127.21.14.124:36603
02:07:17.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=43569
02:07:17.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.14.66
02:07:17.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.14.84:44189
02:07:17.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:07:17.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:07:17.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:07:18.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:18.284215 23219 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:07:18.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:18.284528 23219 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:07:18.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:18.284641 23219 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:07:18.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:18.292999 23219 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:07:18.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:18.293183 23219 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.14.66
02:07:18.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:18.302788 23219 tablet_server_runner.cc:78] Tablet server non-default flags:
02:07:18.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:07:18.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.14.84:44189
02:07:18.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:07:18.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:07:18.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-1/data
02:07:18.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-1/wal
02:07:18.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:07:18.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:07:18.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.14.66:35979
02:07:18.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:07:18.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:07:18.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:07:18.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-1/data/info.pb
02:07:18.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.14.66
02:07:18.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=43569
02:07:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.21.14.126:34291,127.21.14.125:33787,127.21.14.124:36603
02:07:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:07:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:07:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:07:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:07:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:07:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.21.14.66
02:07:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-1/logs
02:07:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:07:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:07:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:07:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:07:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:07:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
02:07:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:07:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:43:30 UTC on bdcb31816ec0
02:07:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12766
02:07:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
02:07:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:18.304802 23219 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:07:18.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:18.306522 23219 file_cache.cc:504] Constructed file cache file cache with capacity 419430
02:07:18.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:18.314919 23227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:18.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:18.314953 23226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:18.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:18.315826 23229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:18.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:18.316489 23219 server_base.cc:1061] running on GCE node
02:07:18.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:18.316965 23219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:07:18.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:18.317849 23219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:07:18.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:18.319051 23219 hybrid_clock.cc:648] HybridClock initialized: now 1783476438319009 us; error 33 us; skew 500 ppm
02:07:18.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:18.319370 23219 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:07:18.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:18.321358 23219 webserver.cc:533] Webserver started at http://127.21.14.66:43569/ using document root <none> and password file <none>
02:07:18.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:18.322036 23219 fs_manager.cc:362] Metadata directory not provided
02:07:18.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:18.322134 23219 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:07:18.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:18.327064 23219 fs_manager.cc:714] Time spent opening directory manager: real 0.003s	user 0.001s	sys 0.004s
02:07:18.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:18.329639 23235 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:07:18.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:18.330838 23219 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
02:07:18.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:18.330957 23219 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-1/data,/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-1/wal
02:07:18.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "34e2861a29784992b9d1a9a5876087a6"
02:07:18.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:06:59 on dist-test-slave-545r"
02:07:18.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:18.331557 23219 fs_report.cc:389] FS layout report
02:07:18.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:07:18.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-1/wal
02:07:18.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-1/wal
02:07:18.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-1/data/data
02:07:18.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:07:18.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:07:18.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:07:18.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:07:18.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:07:18.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:07:18.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:07:18.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:07:18.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:07:18.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:07:18.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:07:18.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:18.351438 23219 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:07:18.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:18.352120 23219 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:07:18.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:18.352339 23219 kserver.cc:163] Server-wide thread pool size limit: 3276
02:07:18.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:18.353168 23219 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:07:18.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:18.354975 23242 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
02:07:18.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:18.360419 23219 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
02:07:18.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:18.360522 23219 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.006s	user 0.001s	sys 0.000s
02:07:18.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:18.360610 23219 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
02:07:18.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:18.363528 23219 ts_tablet_manager.cc:616] Registered 1 tablets
02:07:18.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:18.363608 23219 ts_tablet_manager.cc:595] Time spent register tablets: real 0.003s	user 0.000s	sys 0.000s
02:07:18.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:18.364039 23242 tablet_bootstrap.cc:492] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6: Bootstrap starting.
02:07:18.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:18.387645 23242 log.cc:826] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6: Log is configured to *not* fsync() on all Append() calls
02:07:18.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:18.420238 23242 tablet_bootstrap.cc:492] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6: Bootstrap replayed 1/1 log segments. Stats: ops{read=24 overwritten=0 applied=24 ignored=0} inserts{seen=18 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
02:07:18.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:18.420974 23242 tablet_bootstrap.cc:492] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6: Bootstrap complete.
02:07:18.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:18.421453 23219 rpc_server.cc:307] RPC server started. Bound to: 127.21.14.66:35979
02:07:18.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:18.421519 23349 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.14.66:35979 every 8 connection(s)
02:07:18.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:18.422412 23242 ts_tablet_manager.cc:1403] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6: Time spent bootstrapping tablet: real 0.059s	user 0.037s	sys 0.019s
02:07:18.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:18.422755 23219 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-1/data/info.pb
02:07:18.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:18.430076 21561 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu as pid 23219
02:07:18.431 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:07:18.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:18.441391 23242 raft_consensus.cc:359] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [term 6 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:18.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:18.442360 23242 raft_consensus.cc:740] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34e2861a29784992b9d1a9a5876087a6, State: Initialized, Role: FOLLOWER
02:07:18.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:18.443114 23242 consensus_queue.cc:260] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 24, Last appended: 6.24, Last appended by leader: 24, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:18.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:18.448091 23242 ts_tablet_manager.cc:1434] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6: Time spent starting tablet: real 0.026s	user 0.010s	sys 0.001s
02:07:18.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:18.454026 23351 heartbeater.cc:344] Connected to a master server at 127.21.14.126:34291
02:07:18.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:18.454396 23351 heartbeater.cc:461] Registering TS with master...
02:07:18.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:18.455405 23351 heartbeater.cc:507] Master 127.21.14.126:34291 requested a full tablet report, sending...
02:07:18.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:18.457998 23352 heartbeater.cc:344] Connected to a master server at 127.21.14.125:33787
02:07:18.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:18.458134 23352 heartbeater.cc:461] Registering TS with master...
02:07:18.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:18.458215 21601 ts_manager.cc:194] Re-registered known tserver with Master: 34e2861a29784992b9d1a9a5876087a6 (127.21.14.66:35979)
02:07:18.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:18.458562 23352 heartbeater.cc:507] Master 127.21.14.125:33787 requested a full tablet report, sending...
02:07:18.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:18.459657 21601 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.21.14.66:43121
02:07:18.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:18.459823 21673 ts_manager.cc:194] Re-registered known tserver with Master: 34e2861a29784992b9d1a9a5876087a6 (127.21.14.66:35979)
02:07:18.461 [DEBUG - kudu-nio-6] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 21 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283, tablet=5637f0c18ff54bdd81b35dab09fada4d, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:07:18.462 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.21.14.67:42423
02:07:18.462 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

02:07:18.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:18.462616 21561 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu pid 22856
02:07:18.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:18.465281 23351 heartbeater.cc:499] Master 127.21.14.126:34291 was elected leader, sending a full tablet report...
02:07:18.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:18.471742 23350 heartbeater.cc:344] Connected to a master server at 127.21.14.124:36603
02:07:18.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:18.471863 23350 heartbeater.cc:461] Registering TS with master...
02:07:18.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:18.472117 23350 heartbeater.cc:507] Master 127.21.14.124:36603 requested a full tablet report, sending...
02:07:18.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:18.473428 21745 ts_manager.cc:194] Re-registered known tserver with Master: 34e2861a29784992b9d1a9a5876087a6 (127.21.14.66:35979)
02:07:18.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:18.479621 22982 generic_service.cc:196] Checking for leaks (request via RPC)
02:07:18.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:18.577128 21561 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu with pid 22856
02:07:18.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:18.591755 21659 connection.cc:573] server connection from 127.21.14.67:38059 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
02:07:18.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:18.592072 21590 connection.cc:573] server connection from 127.21.14.67:34819 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
02:07:18.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:18.592269 21729 connection.cc:573] server connection from 127.21.14.67:46495 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
02:07:18.592 [DEBUG - kudu-nio-6] (Connection.java:239) [peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)] handling channelInactive
02:07:18.593 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)] cleaning up while in state READY due to: connection closed
02:07:18.593 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:07:18.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:18.593251 23366 negotiation.cc:336] Failed RPC negotiation. Trace:
02:07:18.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:07:18.576868 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.21.14.67:52617 (local address 127.21.14.66:35979)
02:07:18.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:07:18.577770 (+   902us) server_negotiation.cc:207] Beginning negotiation
02:07:18.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:07:18.577780 (+    10us) server_negotiation.cc:400] Waiting for connection header
02:07:18.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:07:18.591817 (+ 14037us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.14.67:52617: BlockingRecv error: recv got EOF from 127.21.14.67:52617 (error 108)
02:07:18.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":735,"thread_start_us":1783,"threads_started":1}
02:07:18.593 [DEBUG - Time-limited test] (Connection.java:748) [peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)] connecting to peer
02:07:18.594 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:18.594 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:18.594 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423) for tablet 5637f0c18ff54bdd81b35dab09fada4d: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:18.597 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:18.598 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)[L],34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979),790137564e884f989bbd5c36475181af(127.21.14.65:38711)]}]
02:07:18.598 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5637f0c18ff54bdd81b35dab09fada4d, attempt=2, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=4), Traces: [0ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [1ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [2ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [2ms] refreshing cache from master, [2ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [4ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=Deferred@609164854(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007fe67c338998@5fb66faf -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2120ca08 -> wakeup thread Time-limited test)) after lookup
02:07:18.598 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)] connecting to peer
02:07:18.599 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:18.599 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:18.599 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423) for tablet 5637f0c18ff54bdd81b35dab09fada4d: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:18.618 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:18.618 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)[L],34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979),790137564e884f989bbd5c36475181af(127.21.14.65:38711)]}]
02:07:18.618 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5637f0c18ff54bdd81b35dab09fada4d, attempt=4, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=24), Traces: [0ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [1ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [2ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [2ms] refreshing cache from master, [2ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [4ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [5ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [6ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [6ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [22ms] refreshing cache from master, [23ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [24ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=Deferred@609164854(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007fe67c338998@5fb66faf -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@886531667 after retry RPC@107063377), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2120ca08 -> wakeup thread Time-limited test -> (continuation of Deferred@886531667 after retry RPC@107063377))) after lookup
02:07:18.618 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)] connecting to peer
02:07:18.619 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:18.619 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:18.619 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423) for tablet 5637f0c18ff54bdd81b35dab09fada4d: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:18.638 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:18.638 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)[L],34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979),790137564e884f989bbd5c36475181af(127.21.14.65:38711)]}]
02:07:18.638 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5637f0c18ff54bdd81b35dab09fada4d, attempt=6, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=45), Traces: [0ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [1ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [2ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [2ms] refreshing cache from master, [2ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [4ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [5ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [6ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [6ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [22ms] refreshing cache from master, [23ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [24ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [25ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [26ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [26ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [43ms] refreshing cache from master, [43ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [45ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=Deferred@609164854(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007fe67c338998@5fb66faf -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@886531667 after retry RPC@107063377) -> (continuation of Deferred@301856039 after retry RPC@441836649), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2120ca08 -> wakeup thread Time-limited test -> (continuation of Deferred@886531667 after retry RPC@107063377) -> (continuation of Deferred@301856039 after retry RPC@441836649))) after lookup
02:07:18.639 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)] connecting to peer
02:07:18.640 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:18.640 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:18.640 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423) for tablet 5637f0c18ff54bdd81b35dab09fada4d: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:18.758 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:18.759 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)[L],34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979),790137564e884f989bbd5c36475181af(127.21.14.65:38711)]}]
02:07:18.759 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5637f0c18ff54bdd81b35dab09fada4d, attempt=8, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=166), Traces: [0ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [1ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [2ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [2ms] refreshing cache from master, [2ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [4ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [5ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [6ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [6ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [22ms] refreshing cache from master, [23ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [24ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [25ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [26ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [26ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [43ms] refreshing cache from master, [43ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [45ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [46ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [47ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [47ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [163ms] refreshing cache from master, [163ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [165ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=Deferred@609164854(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007fe67c338998@5fb66faf -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@886531667 after retry RPC@107063377) -> (continuation of Deferred@301856039 after retry RPC@441836649) -> (continuation of Deferred@365573213 after retry RPC@537525281), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2120ca08 -> wakeup thread Time-limited test -> (continuation of Deferred@886531667 after retry RPC@107063377) -> (continuation of Deferred@301856039 after retry RPC@441836649) -> (continuation of Deferred@365573213 after retry RPC@537525281))) after lookup
02:07:18.760 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)] connecting to peer
02:07:18.761 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:18.761 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:18.761 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423) for tablet 5637f0c18ff54bdd81b35dab09fada4d: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:19.039 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:19.039 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)[L],34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979),790137564e884f989bbd5c36475181af(127.21.14.65:38711)]}]
02:07:19.039 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5637f0c18ff54bdd81b35dab09fada4d, attempt=10, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=445), Traces: [0ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [1ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [2ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [2ms] refreshing cache from master, [2ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [4ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [5ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [6ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [6ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [22ms] refreshing cache from master, [23ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [24ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [25ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [26ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [26ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [43ms] refreshing cache from master, [43ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [45ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [46ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [47ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [47ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [163ms] refreshing cache from master, [163ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [165ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [166ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [168ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [168ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [443ms] refreshing cache from master, [443ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [445ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=Deferred@609164854(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007fe67c338998@5fb66faf -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@886531667 after retry RPC@107063377) -> (continuation of Deferred@301856039 after retry RPC@441836649) -> (continuation of Deferred@365573213 after retry RPC@537525281) -> (continuation of Deferred@595004827 after retry RPC@1290798368), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2120ca08 -> wakeup thread Time-limited test -> (continuation of Deferred@886531667 after retry RPC@107063377) -> (continuation of Deferred@301856039 after retry RPC@441836649) -> (continuation of Deferred@365573213 after retry RPC@537525281) -> (continuation of Deferred@595004827 after retry RPC@1290798368))) after lookup
02:07:19.040 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)] connecting to peer
02:07:19.041 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:19.041 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:19.041 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423) for tablet 5637f0c18ff54bdd81b35dab09fada4d: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:19.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:19.863996 23367 raft_consensus.cc:493] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1a735d9d94bb4c2e8d28cad5e18d6785)
02:07:19.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:19.864223 23367 raft_consensus.cc:515] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:19.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:19.864948 23367 leader_election.cc:290] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1a735d9d94bb4c2e8d28cad5e18d6785 (127.21.14.67:42423), 34e2861a29784992b9d1a9a5876087a6 (127.21.14.66:35979)
02:07:19.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:19.866837 23055 leader_election.cc:336] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 1a735d9d94bb4c2e8d28cad5e18d6785 (127.21.14.67:42423): Network error: Client connection negotiation failed: client connection to 127.21.14.67:42423: connect: Connection refused (error 111)
02:07:19.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:19.872223 23304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5637f0c18ff54bdd81b35dab09fada4d" candidate_uuid: "790137564e884f989bbd5c36475181af" candidate_term: 8 candidate_status { last_received { term: 7 index: 28 } } ignore_live_leader: false dest_uuid: "34e2861a29784992b9d1a9a5876087a6" is_pre_election: true
02:07:19.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:19.872659 23304 raft_consensus.cc:2468] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 790137564e884f989bbd5c36475181af in term 6.
02:07:19.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:19.873469 23054 leader_election.cc:304] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 34e2861a29784992b9d1a9a5876087a6, 790137564e884f989bbd5c36475181af; no voters: 1a735d9d94bb4c2e8d28cad5e18d6785
02:07:19.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:19.873730 23367 raft_consensus.cc:2804] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 7 FOLLOWER]: Leader pre-election won for term 8
02:07:19.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:19.873842 23367 raft_consensus.cc:493] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 7 FOLLOWER]: Starting leader election (detected failure of leader 1a735d9d94bb4c2e8d28cad5e18d6785)
02:07:19.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:19.873920 23367 raft_consensus.cc:3060] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 7 FOLLOWER]: Advancing to term 8
02:07:19.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:19.875975 23367 raft_consensus.cc:515] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:19.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:19.876492 23367 leader_election.cc:290] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [CANDIDATE]: Term 8 election: Requested vote from peers 1a735d9d94bb4c2e8d28cad5e18d6785 (127.21.14.67:42423), 34e2861a29784992b9d1a9a5876087a6 (127.21.14.66:35979)
02:07:19.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:19.876997 23304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5637f0c18ff54bdd81b35dab09fada4d" candidate_uuid: "790137564e884f989bbd5c36475181af" candidate_term: 8 candidate_status { last_received { term: 7 index: 28 } } ignore_live_leader: false dest_uuid: "34e2861a29784992b9d1a9a5876087a6"
02:07:19.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:19.877194 23304 raft_consensus.cc:3060] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [term 6 FOLLOWER]: Advancing to term 8
02:07:19.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:19.877879 23055 leader_election.cc:336] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 1a735d9d94bb4c2e8d28cad5e18d6785 (127.21.14.67:42423): Network error: Client connection negotiation failed: client connection to 127.21.14.67:42423: connect: Connection refused (error 111)
02:07:19.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:19.880801 23304 raft_consensus.cc:2468] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 790137564e884f989bbd5c36475181af in term 8.
02:07:19.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:19.881145 23054 leader_election.cc:304] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 34e2861a29784992b9d1a9a5876087a6, 790137564e884f989bbd5c36475181af; no voters: 1a735d9d94bb4c2e8d28cad5e18d6785
02:07:19.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:19.881402 23367 raft_consensus.cc:2804] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 8 FOLLOWER]: Leader election won for term 8
02:07:19.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:19.881765 23367 raft_consensus.cc:697] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 8 LEADER]: Becoming Leader. State: Replica: 790137564e884f989bbd5c36475181af, State: Running, Role: LEADER
02:07:19.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:19.882205 23367 consensus_queue.cc:237] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 7.28, Last appended by leader: 28, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:19.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:19.885536 21601 catalog_manager.cc:5697] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af reported cstate change: term changed from 7 to 8, leader changed from 1a735d9d94bb4c2e8d28cad5e18d6785 (127.21.14.67) to 790137564e884f989bbd5c36475181af (127.21.14.65). New cstate: current_term: 8 leader_uuid: "790137564e884f989bbd5c36475181af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } health_report { overall_health: HEALTHY } } }
02:07:20.218 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:20.218 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423),34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979),790137564e884f989bbd5c36475181af(127.21.14.65:38711)[L]]}]
02:07:20.218 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5637f0c18ff54bdd81b35dab09fada4d, attempt=12, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=1625), Traces: [0ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [1ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [2ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [2ms] refreshing cache from master, [2ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [4ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [5ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [6ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [6ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [22ms] refreshing cache from master, [23ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [24ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [25ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [26ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [26ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [43ms] refreshing cache from master, [43ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [45ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [46ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [47ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [47ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [163ms] refreshing cache from master, [163ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [165ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [166ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [168ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [168ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [443ms] refreshing cache from master, [443ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [445ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [446ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [448ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [448ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [1622ms] refreshing cache from master, [1622ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [1625ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=Deferred@609164854(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007fe67c338998@5fb66faf -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@886531667 after retry RPC@107063377) -> (continuation of Deferred@301856039 after retry RPC@441836649) -> (continuation of Deferred@365573213 after retry RPC@537525281) -> (continuation of Deferred@595004827 after retry RPC@1290798368) -> (continuation of Deferred@2099929719 after retry RPC@834646519), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2120ca08 -> wakeup thread Time-limited test -> (continuation of Deferred@886531667 after retry RPC@107063377) -> (continuation of Deferred@301856039 after retry RPC@441836649) -> (continuation of Deferred@365573213 after retry RPC@537525281) -> (continuation of Deferred@595004827 after retry RPC@1290798368) -> (continuation of Deferred@2099929719 after retry RPC@834646519))) after lookup
02:07:20.219 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711)] connecting to peer
02:07:20.220 [DEBUG - kudu-nio-5] (Connection.java:757) [peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711)] Successfully connected to peer
02:07:20.249 [DEBUG - kudu-nio-5] (Negotiator.java:918) Authenticated connection [id: 0x3a29cea9, L:/127.0.0.1:38306 - R:/127.21.14.65:38711] using TOKEN/null
02:07:20.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:20.260308 23304 raft_consensus.cc:1275] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [term 8 FOLLOWER]: Refusing update from remote peer 790137564e884f989bbd5c36475181af: Log matching property violated. Preceding OpId in replica: term: 6 index: 24. Preceding OpId from leader: term: 8 index: 30. (index mismatch)
02:07:20.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:20.260653 23055 consensus_peers.cc:597] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af -> Peer 1a735d9d94bb4c2e8d28cad5e18d6785 (127.21.14.67:42423): Couldn't send request to peer 1a735d9d94bb4c2e8d28cad5e18d6785. Status: Network error: Client connection negotiation failed: client connection to 127.21.14.67:42423: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
02:07:20.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:20.261140 23367 consensus_queue.cc:1048] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [LEADER]: Connected to new peer: Peer: permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 24, Time since last communication: 0.000s
02:07:20.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:20.277776 23385 mvcc.cc:204] Tried to move back new op lower bound from 7305119499296710656 to 7305119497760649216. Current Snapshot: MvccSnapshot[applied={T|T < 7305119499296710656}]
02:07:20.298 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.21.14.67:42423
02:07:20.298 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

02:07:20.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:20.299561 21561 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu
02:07:20.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu
02:07:20.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/wal
02:07:20.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/data
02:07:20.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:07:20.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:07:20.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:07:20.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:07:20.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:07:20.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:07:20.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/logs
02:07:20.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/data/info.pb
02:07:20.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:07:20.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:07:20.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:07:20.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:07:20.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:07:20.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:07:20.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:07:20.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:07:20.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:07:20.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:07:20.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.14.67:42423
02:07:20.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.21.14.67
02:07:20.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.21.14.126:34291,127.21.14.125:33787,127.21.14.124:36603
02:07:20.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=37049
02:07:20.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.14.67
02:07:20.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.14.84:44189
02:07:20.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:07:20.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:07:20.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:07:20.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:20.602116 23394 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:07:20.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:20.602373 23394 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:07:20.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:20.602449 23394 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:07:20.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:20.610277 23394 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:07:20.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:20.610410 23394 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.14.67
02:07:20.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:20.619717 23394 tablet_server_runner.cc:78] Tablet server non-default flags:
02:07:20.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:07:20.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.14.84:44189
02:07:20.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:07:20.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:07:20.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/data
02:07:20.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/wal
02:07:20.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:07:20.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:07:20.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.14.67:42423
02:07:20.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:07:20.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:07:20.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:07:20.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/data/info.pb
02:07:20.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.14.67
02:07:20.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=37049
02:07:20.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.21.14.126:34291,127.21.14.125:33787,127.21.14.124:36603
02:07:20.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:07:20.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:07:20.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:07:20.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:07:20.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:07:20.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.21.14.67
02:07:20.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/logs
02:07:20.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:07:20.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:07:20.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:07:20.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:07:20.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:07:20.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
02:07:20.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:07:20.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:43:30 UTC on bdcb31816ec0
02:07:20.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12766
02:07:20.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
02:07:20.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:20.621238 23394 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:07:20.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:20.622855 23394 file_cache.cc:504] Constructed file cache file cache with capacity 419430
02:07:20.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:20.631345 23400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:20.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:20.631377 23399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:20.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:20.632121 23394 server_base.cc:1061] running on GCE node
02:07:20.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:20.632334 23402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:20.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:20.632928 23394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:07:20.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:20.633728 23394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:07:20.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:20.634922 23394 hybrid_clock.cc:648] HybridClock initialized: now 1783476440634872 us; error 36 us; skew 500 ppm
02:07:20.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:20.635227 23394 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:07:20.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:20.637205 23394 webserver.cc:533] Webserver started at http://127.21.14.67:37049/ using document root <none> and password file <none>
02:07:20.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:20.637825 23394 fs_manager.cc:362] Metadata directory not provided
02:07:20.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:20.637904 23394 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:07:20.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:20.642640 23394 fs_manager.cc:714] Time spent opening directory manager: real 0.003s	user 0.004s	sys 0.000s
02:07:20.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:20.645130 23408 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:07:20.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:20.646255 23394 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
02:07:20.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:20.646375 23394 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/data,/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/wal
02:07:20.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "1a735d9d94bb4c2e8d28cad5e18d6785"
02:07:20.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:06:59 on dist-test-slave-545r"
02:07:20.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:20.646988 23394 fs_report.cc:389] FS layout report
02:07:20.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:07:20.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/wal
02:07:20.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/wal
02:07:20.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/data/data
02:07:20.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:07:20.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:07:20.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:07:20.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:07:20.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:07:20.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:07:20.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:07:20.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:07:20.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:07:20.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:07:20.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:07:20.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:20.682922 23394 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:07:20.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:20.683619 23394 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:07:20.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:20.683887 23394 kserver.cc:163] Server-wide thread pool size limit: 3276
02:07:20.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:20.684697 23394 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:07:20.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:20.686625 23415 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
02:07:20.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:20.692091 23394 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
02:07:20.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:20.692167 23394 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.006s	user 0.001s	sys 0.000s
02:07:20.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:20.692255 23394 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
02:07:20.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:20.695106 23394 ts_tablet_manager.cc:616] Registered 1 tablets
02:07:20.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:20.695187 23394 ts_tablet_manager.cc:595] Time spent register tablets: real 0.003s	user 0.000s	sys 0.000s
02:07:20.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:20.695616 23415 tablet_bootstrap.cc:492] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785: Bootstrap starting.
02:07:20.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:20.719641 23415 log.cc:826] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785: Log is configured to *not* fsync() on all Append() calls
02:07:20.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:20.738729 23394 rpc_server.cc:307] RPC server started. Bound to: 127.21.14.67:42423
02:07:20.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:20.739086 23522 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.14.67:42423 every 8 connection(s)
02:07:20.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:20.740345 23394 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/data/info.pb
02:07:20.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:20.742868 21561 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu as pid 23394
02:07:20.743 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:07:20.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:20.759481 23415 tablet_bootstrap.cc:492] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=21 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
02:07:20.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:20.760113 23415 tablet_bootstrap.cc:492] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785: Bootstrap complete.
02:07:20.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:20.761639 23415 ts_tablet_manager.cc:1403] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785: Time spent bootstrapping tablet: real 0.066s	user 0.037s	sys 0.021s
02:07:20.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:20.768873 23415 raft_consensus.cc:359] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [term 7 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:20.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:20.769841 23415 raft_consensus.cc:740] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a735d9d94bb4c2e8d28cad5e18d6785, State: Initialized, Role: FOLLOWER
02:07:20.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:20.771020 23415 consensus_queue.cc:260] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 7.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:20.812 [DEBUG - kudu-nio-5] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 24 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283, tablet=5637f0c18ff54bdd81b35dab09fada4d, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:07:20.813 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.21.14.65:38711
02:07:20.813 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

02:07:20.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:20.814049 21561 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu pid 23039
02:07:20.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:20.847729 23524 heartbeater.cc:344] Connected to a master server at 127.21.14.126:34291
02:07:20.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:20.847838 23523 heartbeater.cc:344] Connected to a master server at 127.21.14.124:36603
02:07:20.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:20.848165 23523 heartbeater.cc:461] Registering TS with master...
02:07:20.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:20.848171 23524 heartbeater.cc:461] Registering TS with master...
02:07:20.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:20.849438 23524 heartbeater.cc:507] Master 127.21.14.126:34291 requested a full tablet report, sending...
02:07:20.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:20.851979 23523 heartbeater.cc:507] Master 127.21.14.124:36603 requested a full tablet report, sending...
02:07:20.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:20.852192 23415 ts_tablet_manager.cc:1434] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785: Time spent starting tablet: real 0.090s	user 0.024s	sys 0.028s
02:07:20.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:20.853322 21601 ts_manager.cc:194] Re-registered known tserver with Master: 1a735d9d94bb4c2e8d28cad5e18d6785 (127.21.14.67:42423)
02:07:20.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:20.854769 21601 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.21.14.67:52529
02:07:20.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:20.857139 21745 ts_manager.cc:194] Re-registered known tserver with Master: 1a735d9d94bb4c2e8d28cad5e18d6785 (127.21.14.67:42423)
02:07:20.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:20.858068 23524 heartbeater.cc:499] Master 127.21.14.126:34291 was elected leader, sending a full tablet report...
02:07:20.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:20.861948 23163 generic_service.cc:196] Checking for leaks (request via RPC)
02:07:20.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:20.862398 23527 heartbeater.cc:344] Connected to a master server at 127.21.14.125:33787
02:07:20.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:20.864754 23527 heartbeater.cc:461] Registering TS with master...
02:07:20.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:20.864979 23527 heartbeater.cc:507] Master 127.21.14.125:33787 requested a full tablet report, sending...
02:07:20.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:20.866151 21673 ts_manager.cc:194] Re-registered known tserver with Master: 1a735d9d94bb4c2e8d28cad5e18d6785 (127.21.14.67:42423)
02:07:20.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:20.959017 21561 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu with pid 23039
02:07:20.979 [DEBUG - kudu-nio-5] (Connection.java:239) [peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711)] handling channelInactive
02:07:20.980 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711)] cleaning up while in state READY due to: connection closed
02:07:20.981 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:07:20.981 [DEBUG - Time-limited test] (Connection.java:748) [peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711)] connecting to peer
02:07:20.982 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711
02:07:20.982 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711)] cleaning up while in state CONNECTING due to: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711
02:07:20.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:20.982344 23537 negotiation.cc:336] Failed RPC negotiation. Trace:
02:07:20.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:07:20.843646 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.21.14.65:40203 (local address 127.21.14.67:42423)
02:07:20.983 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 790137564e884f989bbd5c36475181af(127.21.14.65:38711) for tablet 5637f0c18ff54bdd81b35dab09fada4d: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711
02:07:20.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:07:20.850031 (+  6385us) server_negotiation.cc:207] Beginning negotiation
02:07:20.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:07:20.850040 (+     9us) server_negotiation.cc:400] Waiting for connection header
02:07:20.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:07:20.850072 (+    32us) server_negotiation.cc:408] Connection header received
02:07:20.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:07:20.850185 (+   113us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
02:07:20.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:07:20.850192 (+     7us) server_negotiation.cc:462] Received NEGOTIATE request from client
02:07:20.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:07:20.850434 (+   242us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
02:07:20.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:07:20.850593 (+   159us) server_negotiation.cc:227] Negotiated authn=SASL
02:07:20.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:07:20.854056 (+  3463us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
02:07:20.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:07:20.855857 (+  1801us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
02:07:20.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:07:20.958169 (+102312us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
02:07:20.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:07:20.958745 (+   576us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
02:07:20.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:07:20.966944 (+  8199us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
02:07:20.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:07:20.967039 (+    95us) server_negotiation.cc:882] Received SASL_INITIATE request from client
02:07:20.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:07:20.967056 (+    17us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
02:07:20.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:07:20.967664 (+   608us) server_negotiation.cc:923] calling sasl_server_start()
02:07:20.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:07:20.967713 (+    49us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
02:07:20.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:07:20.967734 (+    21us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
02:07:20.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:07:20.967823 (+    89us) server_negotiation.cc:1036] Waiting for connection context
02:07:20.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:07:20.979694 (+ 11871us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.14.65:40203: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
02:07:20.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":6257,"thread_start_us":17917,"threads_started":1}
02:07:21.000 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:21.000 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423),34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979),790137564e884f989bbd5c36475181af(127.21.14.65:38711)[L]]}]
02:07:21.001 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5637f0c18ff54bdd81b35dab09fada4d, attempt=2, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=19), Traces: [0ms] sending RPC to server 790137564e884f989bbd5c36475181af, [1ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [2ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [19ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=Deferred@1396496412(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007fe67c338998@1fc0fb85 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7b126e33 -> wakeup thread Time-limited test)) after lookup
02:07:21.001 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711)] connecting to peer
02:07:21.003 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711
02:07:21.003 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711)] cleaning up while in state CONNECTING due to: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711
02:07:21.003 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 790137564e884f989bbd5c36475181af(127.21.14.65:38711) for tablet 5637f0c18ff54bdd81b35dab09fada4d: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711
02:07:21.018 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:21.019 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423),34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979),790137564e884f989bbd5c36475181af(127.21.14.65:38711)[L]]}]
02:07:21.019 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5637f0c18ff54bdd81b35dab09fada4d, attempt=4, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=38), Traces: [0ms] sending RPC to server 790137564e884f989bbd5c36475181af, [1ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [2ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [19ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [20ms] sending RPC to server 790137564e884f989bbd5c36475181af, [22ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [22ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [34ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [37ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=Deferred@1396496412(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007fe67c338998@1fc0fb85 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1639390730 after retry RPC@346737029), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7b126e33 -> wakeup thread Time-limited test -> (continuation of Deferred@1639390730 after retry RPC@346737029))) after lookup
02:07:21.019 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711)] connecting to peer
02:07:21.021 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711
02:07:21.021 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711)] cleaning up while in state CONNECTING due to: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711
02:07:21.021 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 790137564e884f989bbd5c36475181af(127.21.14.65:38711) for tablet 5637f0c18ff54bdd81b35dab09fada4d: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711
02:07:21.039 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:21.039 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423),34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979),790137564e884f989bbd5c36475181af(127.21.14.65:38711)[L]]}]
02:07:21.039 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5637f0c18ff54bdd81b35dab09fada4d, attempt=6, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=58), Traces: [0ms] sending RPC to server 790137564e884f989bbd5c36475181af, [1ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [2ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [19ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [20ms] sending RPC to server 790137564e884f989bbd5c36475181af, [22ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [22ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [34ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [37ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [38ms] sending RPC to server 790137564e884f989bbd5c36475181af, [40ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [40ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [55ms] refreshing cache from master, [55ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [57ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=Deferred@1396496412(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007fe67c338998@1fc0fb85 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1639390730 after retry RPC@346737029) -> (continuation of Deferred@334971292 after retry RPC@858089097), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7b126e33 -> wakeup thread Time-limited test -> (continuation of Deferred@1639390730 after retry RPC@346737029) -> (continuation of Deferred@334971292 after retry RPC@858089097))) after lookup
02:07:21.040 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711)] connecting to peer
02:07:21.041 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711
02:07:21.041 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711)] cleaning up while in state CONNECTING due to: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711
02:07:21.042 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 790137564e884f989bbd5c36475181af(127.21.14.65:38711) for tablet 5637f0c18ff54bdd81b35dab09fada4d: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711
02:07:21.139 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:21.139 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423),34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979),790137564e884f989bbd5c36475181af(127.21.14.65:38711)[L]]}]
02:07:21.139 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5637f0c18ff54bdd81b35dab09fada4d, attempt=8, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=158), Traces: [0ms] sending RPC to server 790137564e884f989bbd5c36475181af, [1ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [2ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [19ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [20ms] sending RPC to server 790137564e884f989bbd5c36475181af, [22ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [22ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [34ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [37ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [38ms] sending RPC to server 790137564e884f989bbd5c36475181af, [40ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [40ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [55ms] refreshing cache from master, [55ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [57ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [59ms] sending RPC to server 790137564e884f989bbd5c36475181af, [61ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [61ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [155ms] refreshing cache from master, [155ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [158ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=Deferred@1396496412(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007fe67c338998@1fc0fb85 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1639390730 after retry RPC@346737029) -> (continuation of Deferred@334971292 after retry RPC@858089097) -> (continuation of Deferred@2024734508 after retry RPC@1613221790), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7b126e33 -> wakeup thread Time-limited test -> (continuation of Deferred@1639390730 after retry RPC@346737029) -> (continuation of Deferred@334971292 after retry RPC@858089097) -> (continuation of Deferred@2024734508 after retry RPC@1613221790))) after lookup
02:07:21.140 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711)] connecting to peer
02:07:21.141 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711
02:07:21.141 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711)] cleaning up while in state CONNECTING due to: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711
02:07:21.141 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 790137564e884f989bbd5c36475181af(127.21.14.65:38711) for tablet 5637f0c18ff54bdd81b35dab09fada4d: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711
02:07:21.238 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:21.238 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423),34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979),790137564e884f989bbd5c36475181af(127.21.14.65:38711)[L]]}]
02:07:21.239 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5637f0c18ff54bdd81b35dab09fada4d, attempt=10, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=257), Traces: [0ms] sending RPC to server 790137564e884f989bbd5c36475181af, [1ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [2ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [19ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [20ms] sending RPC to server 790137564e884f989bbd5c36475181af, [22ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [22ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [34ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [37ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [38ms] sending RPC to server 790137564e884f989bbd5c36475181af, [40ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [40ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [55ms] refreshing cache from master, [55ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [57ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [59ms] sending RPC to server 790137564e884f989bbd5c36475181af, [61ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [61ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [155ms] refreshing cache from master, [155ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [158ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [159ms] sending RPC to server 790137564e884f989bbd5c36475181af, [160ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [160ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [254ms] refreshing cache from master, [254ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [257ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=Deferred@1396496412(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007fe67c338998@1fc0fb85 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1639390730 after retry RPC@346737029) -> (continuation of Deferred@334971292 after retry RPC@858089097) -> (continuation of Deferred@2024734508 after retry RPC@1613221790) -> (continuation of Deferred@1563139681 after retry RPC@61796709), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7b126e33 -> wakeup thread Time-limited test -> (continuation of Deferred@1639390730 after retry RPC@346737029) -> (continuation of Deferred@334971292 after retry RPC@858089097) -> (continuation of Deferred@2024734508 after retry RPC@1613221790) -> (continuation of Deferred@1563139681 after retry RPC@61796709))) after lookup
02:07:21.239 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711)] connecting to peer
02:07:21.240 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711
02:07:21.240 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711)] cleaning up while in state CONNECTING due to: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711
02:07:21.241 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 790137564e884f989bbd5c36475181af(127.21.14.65:38711) for tablet 5637f0c18ff54bdd81b35dab09fada4d: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711
02:07:22.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:22.101332 23540 raft_consensus.cc:493] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
02:07:22.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:22.101605 23540 raft_consensus.cc:515] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:22.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:22.103242 23540 leader_election.cc:290] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 34e2861a29784992b9d1a9a5876087a6 (127.21.14.66:35979), 790137564e884f989bbd5c36475181af (127.21.14.65:38711)
02:07:22.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:22.104722 23410 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.14.65:38711: connect: Connection refused (error 111)
02:07:22.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:22.106217 23410 leader_election.cc:336] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 790137564e884f989bbd5c36475181af (127.21.14.65:38711): Network error: Client connection negotiation failed: client connection to 127.21.14.65:38711: connect: Connection refused (error 111)
02:07:22.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:22.109494 23304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5637f0c18ff54bdd81b35dab09fada4d" candidate_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" candidate_term: 8 candidate_status { last_received { term: 7 index: 28 } } ignore_live_leader: false dest_uuid: "34e2861a29784992b9d1a9a5876087a6" is_pre_election: true
02:07:22.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:22.110236 23409 leader_election.cc:304] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1a735d9d94bb4c2e8d28cad5e18d6785; no voters: 34e2861a29784992b9d1a9a5876087a6, 790137564e884f989bbd5c36475181af
02:07:22.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:22.110550 23540 raft_consensus.cc:2749] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
02:07:22.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:22.458918 23545 raft_consensus.cc:493] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 790137564e884f989bbd5c36475181af)
02:07:22.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:22.459115 23545 raft_consensus.cc:515] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:22.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:22.460601 23545 leader_election.cc:290] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1a735d9d94bb4c2e8d28cad5e18d6785 (127.21.14.67:42423), 790137564e884f989bbd5c36475181af (127.21.14.65:38711)
02:07:22.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:22.462468 23237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.14.65:38711: connect: Connection refused (error 111)
02:07:22.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:22.464188 23237 leader_election.cc:336] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 790137564e884f989bbd5c36475181af (127.21.14.65:38711): Network error: Client connection negotiation failed: client connection to 127.21.14.65:38711: connect: Connection refused (error 111)
02:07:22.468 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:22.468178 23477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5637f0c18ff54bdd81b35dab09fada4d" candidate_uuid: "34e2861a29784992b9d1a9a5876087a6" candidate_term: 9 candidate_status { last_received { term: 8 index: 32 } } ignore_live_leader: false dest_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" is_pre_election: true
02:07:22.468 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:22.468541 23477 raft_consensus.cc:2468] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34e2861a29784992b9d1a9a5876087a6 in term 7.
02:07:22.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:22.469406 23237 leader_election.cc:304] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1a735d9d94bb4c2e8d28cad5e18d6785, 34e2861a29784992b9d1a9a5876087a6; no voters: 790137564e884f989bbd5c36475181af
02:07:22.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:22.469774 23545 raft_consensus.cc:2804] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [term 8 FOLLOWER]: Leader pre-election won for term 9
02:07:22.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:22.469873 23545 raft_consensus.cc:493] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 790137564e884f989bbd5c36475181af)
02:07:22.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:22.469938 23545 raft_consensus.cc:3060] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [term 8 FOLLOWER]: Advancing to term 9
02:07:22.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:22.471935 23545 raft_consensus.cc:515] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:22.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:22.472410 23545 leader_election.cc:290] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [CANDIDATE]: Term 9 election: Requested vote from peers 1a735d9d94bb4c2e8d28cad5e18d6785 (127.21.14.67:42423), 790137564e884f989bbd5c36475181af (127.21.14.65:38711)
02:07:22.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:22.472872 23477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5637f0c18ff54bdd81b35dab09fada4d" candidate_uuid: "34e2861a29784992b9d1a9a5876087a6" candidate_term: 9 candidate_status { last_received { term: 8 index: 32 } } ignore_live_leader: false dest_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785"
02:07:22.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:22.473070 23477 raft_consensus.cc:3060] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [term 7 FOLLOWER]: Advancing to term 9
02:07:22.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:22.473888 23237 leader_election.cc:336] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 790137564e884f989bbd5c36475181af (127.21.14.65:38711): Network error: Client connection negotiation failed: client connection to 127.21.14.65:38711: connect: Connection refused (error 111)
02:07:22.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:22.476958 23477 raft_consensus.cc:2468] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34e2861a29784992b9d1a9a5876087a6 in term 9.
02:07:22.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:22.477357 23237 leader_election.cc:304] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1a735d9d94bb4c2e8d28cad5e18d6785, 34e2861a29784992b9d1a9a5876087a6; no voters: 790137564e884f989bbd5c36475181af
02:07:22.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:22.477622 23545 raft_consensus.cc:2804] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [term 9 FOLLOWER]: Leader election won for term 9
02:07:22.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:22.478010 23545 raft_consensus.cc:697] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [term 9 LEADER]: Becoming Leader. State: Replica: 34e2861a29784992b9d1a9a5876087a6, State: Running, Role: LEADER
02:07:22.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:22.478482 23545 consensus_queue.cc:237] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 8.32, Last appended by leader: 32, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:22.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:22.482038 21601 catalog_manager.cc:5697] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 reported cstate change: term changed from 8 to 9, leader changed from 790137564e884f989bbd5c36475181af (127.21.14.65) to 34e2861a29784992b9d1a9a5876087a6 (127.21.14.66). New cstate: current_term: 9 leader_uuid: "34e2861a29784992b9d1a9a5876087a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } health_report { overall_health: UNKNOWN } } }
02:07:22.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:22.924520 23477 raft_consensus.cc:1275] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [term 9 FOLLOWER]: Refusing update from remote peer 34e2861a29784992b9d1a9a5876087a6: Log matching property violated. Preceding OpId in replica: term: 7 index: 28. Preceding OpId from leader: term: 9 index: 33. (index mismatch)
02:07:22.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:22.925297 23545 consensus_queue.cc:1048] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 28, Time since last communication: 0.000s
02:07:22.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:22.937795 23237 consensus_peers.cc:597] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 -> Peer 790137564e884f989bbd5c36475181af (127.21.14.65:38711): Couldn't send request to peer 790137564e884f989bbd5c36475181af. Status: Network error: Client connection negotiation failed: client connection to 127.21.14.65:38711: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
02:07:23.259 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:23.259 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423),34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)[L],790137564e884f989bbd5c36475181af(127.21.14.65:38711)]}]
02:07:23.260 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5637f0c18ff54bdd81b35dab09fada4d, attempt=12, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=2278), Traces: [0ms] sending RPC to server 790137564e884f989bbd5c36475181af, [1ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [2ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [19ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [20ms] sending RPC to server 790137564e884f989bbd5c36475181af, [22ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [22ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [34ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [37ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [38ms] sending RPC to server 790137564e884f989bbd5c36475181af, [40ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [40ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [55ms] refreshing cache from master, [55ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [57ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [59ms] sending RPC to server 790137564e884f989bbd5c36475181af, [61ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [61ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [155ms] refreshing cache from master, [155ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [158ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [159ms] sending RPC to server 790137564e884f989bbd5c36475181af, [160ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [160ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [254ms] refreshing cache from master, [254ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [257ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [258ms] sending RPC to server 790137564e884f989bbd5c36475181af, [259ms] received response from server 790137564e884f989bbd5c36475181af: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [260ms] delaying RPC due to: Network error: Failed to connect to peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711): Connection refused: /127.21.14.65:38711, [2275ms] refreshing cache from master, [2275ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [2278ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=Deferred@1396496412(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007fe67c338998@1fc0fb85 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1639390730 after retry RPC@346737029) -> (continuation of Deferred@334971292 after retry RPC@858089097) -> (continuation of Deferred@2024734508 after retry RPC@1613221790) -> (continuation of Deferred@1563139681 after retry RPC@61796709) -> (continuation of Deferred@952973387 after retry RPC@1276953582), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7b126e33 -> wakeup thread Time-limited test -> (continuation of Deferred@1639390730 after retry RPC@346737029) -> (continuation of Deferred@334971292 after retry RPC@858089097) -> (continuation of Deferred@2024734508 after retry RPC@1613221790) -> (continuation of Deferred@1563139681 after retry RPC@61796709) -> (continuation of Deferred@952973387 after retry RPC@1276953582))) after lookup
02:07:23.260 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)] connecting to peer
02:07:23.261 [DEBUG - kudu-nio-4] (Connection.java:757) [peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)] Successfully connected to peer
02:07:23.288 [DEBUG - kudu-nio-4] (Negotiator.java:918) Authenticated connection [id: 0x21706b6c, L:/127.0.0.1:32986 - R:/127.21.14.66:35979] using TOKEN/null
02:07:23.318 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.21.14.65:38711
02:07:23.318 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

02:07:23.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:23.318948 21561 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu
02:07:23.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu
02:07:23.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-0/wal
02:07:23.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-0/data
02:07:23.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:07:23.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:07:23.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:07:23.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:07:23.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:07:23.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:07:23.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-0/logs
02:07:23.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-0/data/info.pb
02:07:23.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:07:23.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:07:23.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:07:23.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:07:23.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:07:23.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:07:23.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:07:23.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:07:23.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:07:23.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:07:23.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.14.65:38711
02:07:23.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.21.14.65
02:07:23.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.21.14.126:34291,127.21.14.125:33787,127.21.14.124:36603
02:07:23.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=36395
02:07:23.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.14.65
02:07:23.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.14.84:44189
02:07:23.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:07:23.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:07:23.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:07:23.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:23.622692 23572 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:07:23.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:23.622957 23572 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:07:23.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:23.623047 23572 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:07:23.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:23.631018 23572 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:07:23.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:23.631162 23572 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.14.65
02:07:23.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:23.640604 23572 tablet_server_runner.cc:78] Tablet server non-default flags:
02:07:23.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:07:23.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.14.84:44189
02:07:23.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:07:23.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:07:23.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-0/data
02:07:23.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-0/wal
02:07:23.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:07:23.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:07:23.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.14.65:38711
02:07:23.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:07:23.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:07:23.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:07:23.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-0/data/info.pb
02:07:23.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.14.65
02:07:23.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=36395
02:07:23.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.21.14.126:34291,127.21.14.125:33787,127.21.14.124:36603
02:07:23.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:07:23.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:07:23.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:07:23.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:07:23.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:07:23.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.21.14.65
02:07:23.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-0/logs
02:07:23.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:07:23.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:07:23.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:07:23.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:07:23.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:07:23.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
02:07:23.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:07:23.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:43:30 UTC on bdcb31816ec0
02:07:23.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12766
02:07:23.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
02:07:23.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:23.642053 23572 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:07:23.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:23.643672 23572 file_cache.cc:504] Constructed file cache file cache with capacity 419430
02:07:23.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:23.652562 23579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:23.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:23.652619 23582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:23.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:23.652599 23580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:23.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:23.653759 23572 server_base.cc:1061] running on GCE node
02:07:23.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:23.654276 23572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:07:23.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:23.655063 23572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:07:23.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:23.656267 23572 hybrid_clock.cc:648] HybridClock initialized: now 1783476443656213 us; error 47 us; skew 500 ppm
02:07:23.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:23.656602 23572 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:07:23.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:23.658516 23572 webserver.cc:533] Webserver started at http://127.21.14.65:36395/ using document root <none> and password file <none>
02:07:23.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:23.659148 23572 fs_manager.cc:362] Metadata directory not provided
02:07:23.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:23.659229 23572 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:07:23.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:23.664077 23572 fs_manager.cc:714] Time spent opening directory manager: real 0.003s	user 0.003s	sys 0.000s
02:07:23.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:23.666543 23588 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:07:23.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:23.667821 23572 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.002s
02:07:23.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:23.667944 23572 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-0/data,/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-0/wal
02:07:23.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "790137564e884f989bbd5c36475181af"
02:07:23.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:06:59 on dist-test-slave-545r"
02:07:23.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:23.668608 23572 fs_report.cc:389] FS layout report
02:07:23.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:07:23.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-0/wal
02:07:23.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-0/wal
02:07:23.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-0/data/data
02:07:23.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:07:23.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:07:23.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:07:23.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:07:23.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:07:23.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:07:23.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:07:23.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:07:23.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:07:23.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:07:23.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:07:23.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:23.683032 23572 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:07:23.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:23.683749 23572 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:07:23.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:23.683992 23572 kserver.cc:163] Server-wide thread pool size limit: 3276
02:07:23.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:23.684837 23572 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:07:23.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:23.686738 23595 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
02:07:23.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:23.692628 23572 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
02:07:23.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:23.692706 23572 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.006s	user 0.001s	sys 0.000s
02:07:23.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:23.692796 23572 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
02:07:23.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:23.695595 23572 ts_tablet_manager.cc:616] Registered 1 tablets
02:07:23.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:23.695688 23572 ts_tablet_manager.cc:595] Time spent register tablets: real 0.003s	user 0.000s	sys 0.000s
02:07:23.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:23.696086 23595 tablet_bootstrap.cc:492] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af: Bootstrap starting.
02:07:23.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:23.721767 23595 log.cc:826] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af: Log is configured to *not* fsync() on all Append() calls
02:07:23.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:23.732515 23572 rpc_server.cc:307] RPC server started. Bound to: 127.21.14.65:38711
02:07:23.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:23.732560 23702 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.14.65:38711 every 8 connection(s)
02:07:23.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:23.734088 23572 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-0/data/info.pb
02:07:23.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:23.743489 21561 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu as pid 23572
02:07:23.744 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:07:23.781 [DEBUG - kudu-nio-4] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 27 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283, tablet=5637f0c18ff54bdd81b35dab09fada4d, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:07:23.781 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.21.14.66:35979
02:07:23.781 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

02:07:23.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:23.781340 23595 tablet_bootstrap.cc:492] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=24 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
02:07:23.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:23.781957 23595 tablet_bootstrap.cc:492] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af: Bootstrap complete.
02:07:23.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:23.782378 21561 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu pid 23219
02:07:23.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:23.787266 23595 ts_tablet_manager.cc:1403] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af: Time spent bootstrapping tablet: real 0.092s	user 0.054s	sys 0.011s
02:07:23.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:23.790961 23705 heartbeater.cc:344] Connected to a master server at 127.21.14.125:33787
02:07:23.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:23.791296 23705 heartbeater.cc:461] Registering TS with master...
02:07:23.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:23.792291 23705 heartbeater.cc:507] Master 127.21.14.125:33787 requested a full tablet report, sending...
02:07:23.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:23.793211 23595 raft_consensus.cc:359] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 8 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:23.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:23.794205 23595 raft_consensus.cc:740] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: 790137564e884f989bbd5c36475181af, State: Initialized, Role: FOLLOWER
02:07:23.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:23.794921 23595 consensus_queue.cc:260] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 8.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:23.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:23.795579 21673 ts_manager.cc:194] Re-registered known tserver with Master: 790137564e884f989bbd5c36475181af (127.21.14.65:38711)
02:07:23.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:23.796746 23595 ts_tablet_manager.cc:1434] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af: Time spent starting tablet: real 0.009s	user 0.009s	sys 0.001s
02:07:23.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:23.797343 23703 heartbeater.cc:344] Connected to a master server at 127.21.14.124:36603
02:07:23.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:23.797499 23703 heartbeater.cc:461] Registering TS with master...
02:07:23.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:23.798002 23703 heartbeater.cc:507] Master 127.21.14.124:36603 requested a full tablet report, sending...
02:07:23.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:23.798892 23345 generic_service.cc:196] Checking for leaks (request via RPC)
02:07:23.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:23.799113 21745 ts_manager.cc:194] Re-registered known tserver with Master: 790137564e884f989bbd5c36475181af (127.21.14.65:38711)
02:07:23.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:23.800396 23704 heartbeater.cc:344] Connected to a master server at 127.21.14.126:34291
02:07:23.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:23.800531 23704 heartbeater.cc:461] Registering TS with master...
02:07:23.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:23.800945 23704 heartbeater.cc:507] Master 127.21.14.126:34291 requested a full tablet report, sending...
02:07:23.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:23.802026 21601 ts_manager.cc:194] Re-registered known tserver with Master: 790137564e884f989bbd5c36475181af (127.21.14.65:38711)
02:07:23.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:23.803100 21601 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.21.14.65:35291
02:07:23.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:23.805217 23704 heartbeater.cc:499] Master 127.21.14.126:34291 was elected leader, sending a full tablet report...
02:07:23.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:23.896510 21561 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu with pid 23219
02:07:23.910 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:07:23.911 [DEBUG - kudu-nio-4] (Connection.java:239) [peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)] handling channelInactive
02:07:23.911 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)] cleaning up while in state READY due to: connection closed
02:07:23.916 [DEBUG - pool-2-thread-1] (Connection.java:748) [peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)] connecting to peer
02:07:23.917 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979
02:07:23.917 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)] cleaning up while in state CONNECTING due to: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979
02:07:23.917 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979) for tablet 5637f0c18ff54bdd81b35dab09fada4d: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979
02:07:23.938 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:23.938 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423),34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)[L],790137564e884f989bbd5c36475181af(127.21.14.65:38711)]}]
02:07:23.938 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5637f0c18ff54bdd81b35dab09fada4d, attempt=3, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=27), Traces: [0ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [0ms] delaying RPC due to: Illegal state: connection is terminated, [4ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [6ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [6ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [24ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [27ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=Deferred@597192725(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007fe67c338998@c8a05c8 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1e05f97 -> wakeup thread Time-limited test)) after lookup
02:07:23.939 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)] connecting to peer
02:07:23.940 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979
02:07:23.940 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)] cleaning up while in state CONNECTING due to: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979
02:07:23.940 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979) for tablet 5637f0c18ff54bdd81b35dab09fada4d: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979
02:07:23.958 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:23.958 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423),34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)[L],790137564e884f989bbd5c36475181af(127.21.14.65:38711)]}]
02:07:23.958 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5637f0c18ff54bdd81b35dab09fada4d, attempt=5, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=47), Traces: [0ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [0ms] delaying RPC due to: Illegal state: connection is terminated, [4ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [6ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [6ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [24ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [27ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [28ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [29ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [29ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [45ms] refreshing cache from master, [45ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [47ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=Deferred@597192725(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007fe67c338998@c8a05c8 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1475011386 after retry RPC@1416954551), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1e05f97 -> wakeup thread Time-limited test -> (continuation of Deferred@1475011386 after retry RPC@1416954551))) after lookup
02:07:23.959 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)] connecting to peer
02:07:23.960 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979
02:07:23.960 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)] cleaning up while in state CONNECTING due to: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979
02:07:23.960 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979) for tablet 5637f0c18ff54bdd81b35dab09fada4d: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979
02:07:24.039 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:24.039 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423),34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)[L],790137564e884f989bbd5c36475181af(127.21.14.65:38711)]}]
02:07:24.039 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5637f0c18ff54bdd81b35dab09fada4d, attempt=7, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=128), Traces: [0ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [0ms] delaying RPC due to: Illegal state: connection is terminated, [4ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [6ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [6ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [24ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [27ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [28ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [29ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [29ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [45ms] refreshing cache from master, [45ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [47ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [48ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [49ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [49ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [125ms] refreshing cache from master, [125ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [127ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=Deferred@597192725(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007fe67c338998@c8a05c8 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1475011386 after retry RPC@1416954551) -> (continuation of Deferred@1924469695 after retry RPC@237623268), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1e05f97 -> wakeup thread Time-limited test -> (continuation of Deferred@1475011386 after retry RPC@1416954551) -> (continuation of Deferred@1924469695 after retry RPC@237623268))) after lookup
02:07:24.040 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)] connecting to peer
02:07:24.040 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979
02:07:24.040 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)] cleaning up while in state CONNECTING due to: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979
02:07:24.041 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979) for tablet 5637f0c18ff54bdd81b35dab09fada4d: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979
02:07:24.078 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:24.078 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423),34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)[L],790137564e884f989bbd5c36475181af(127.21.14.65:38711)]}]
02:07:24.079 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5637f0c18ff54bdd81b35dab09fada4d, attempt=9, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=167), Traces: [0ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [0ms] delaying RPC due to: Illegal state: connection is terminated, [4ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [6ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [6ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [24ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [27ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [28ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [29ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [29ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [45ms] refreshing cache from master, [45ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [47ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [48ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [49ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [49ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [125ms] refreshing cache from master, [125ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [127ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [128ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [129ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [130ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [164ms] refreshing cache from master, [164ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [167ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=Deferred@597192725(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007fe67c338998@c8a05c8 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1475011386 after retry RPC@1416954551) -> (continuation of Deferred@1924469695 after retry RPC@237623268) -> (continuation of Deferred@1179651198 after retry RPC@518366483), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1e05f97 -> wakeup thread Time-limited test -> (continuation of Deferred@1475011386 after retry RPC@1416954551) -> (continuation of Deferred@1924469695 after retry RPC@237623268) -> (continuation of Deferred@1179651198 after retry RPC@518366483))) after lookup
02:07:24.079 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)] connecting to peer
02:07:24.080 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979
02:07:24.080 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)] cleaning up while in state CONNECTING due to: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979
02:07:24.080 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979) for tablet 5637f0c18ff54bdd81b35dab09fada4d: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979
02:07:24.178 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:24.178 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423),34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)[L],790137564e884f989bbd5c36475181af(127.21.14.65:38711)]}]
02:07:24.179 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5637f0c18ff54bdd81b35dab09fada4d, attempt=11, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=267), Traces: [0ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [0ms] delaying RPC due to: Illegal state: connection is terminated, [4ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [6ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [6ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [24ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [27ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [28ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [29ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [29ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [45ms] refreshing cache from master, [45ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [47ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [48ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [49ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [49ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [125ms] refreshing cache from master, [125ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [127ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [128ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [129ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [130ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [164ms] refreshing cache from master, [164ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [167ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [168ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [169ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [169ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [264ms] refreshing cache from master, [264ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [267ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=Deferred@597192725(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007fe67c338998@c8a05c8 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1475011386 after retry RPC@1416954551) -> (continuation of Deferred@1924469695 after retry RPC@237623268) -> (continuation of Deferred@1179651198 after retry RPC@518366483) -> (continuation of Deferred@734728762 after retry RPC@414578541), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1e05f97 -> wakeup thread Time-limited test -> (continuation of Deferred@1475011386 after retry RPC@1416954551) -> (continuation of Deferred@1924469695 after retry RPC@237623268) -> (continuation of Deferred@1179651198 after retry RPC@518366483) -> (continuation of Deferred@734728762 after retry RPC@414578541))) after lookup
02:07:24.179 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)] connecting to peer
02:07:24.180 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979
02:07:24.180 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)] cleaning up while in state CONNECTING due to: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979
02:07:24.180 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979) for tablet 5637f0c18ff54bdd81b35dab09fada4d: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979
02:07:25.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:25.113772 23717 raft_consensus.cc:493] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
02:07:25.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:25.114018 23717 raft_consensus.cc:515] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:25.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:25.115743 23717 leader_election.cc:290] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1a735d9d94bb4c2e8d28cad5e18d6785 (127.21.14.67:42423), 34e2861a29784992b9d1a9a5876087a6 (127.21.14.66:35979)
02:07:25.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:25.119728 23589 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.14.66:35979: connect: Connection refused (error 111)
02:07:25.121 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:25.121275 23589 leader_election.cc:336] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 34e2861a29784992b9d1a9a5876087a6 (127.21.14.66:35979): Network error: Client connection negotiation failed: client connection to 127.21.14.66:35979: connect: Connection refused (error 111)
02:07:25.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:25.124347 23477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5637f0c18ff54bdd81b35dab09fada4d" candidate_uuid: "790137564e884f989bbd5c36475181af" candidate_term: 9 candidate_status { last_received { term: 8 index: 32 } } ignore_live_leader: false dest_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" is_pre_election: true
02:07:25.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:25.125164 23590 leader_election.cc:304] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 790137564e884f989bbd5c36475181af; no voters: 1a735d9d94bb4c2e8d28cad5e18d6785, 34e2861a29784992b9d1a9a5876087a6
02:07:25.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:25.125502 23717 raft_consensus.cc:2749] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
02:07:25.219 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:25.219 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423),34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)[L],790137564e884f989bbd5c36475181af(127.21.14.65:38711)]}]
02:07:25.219 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5637f0c18ff54bdd81b35dab09fada4d, attempt=13, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=1308), Traces: [0ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [0ms] delaying RPC due to: Illegal state: connection is terminated, [4ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [6ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [6ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [24ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [27ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [28ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [29ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [29ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [45ms] refreshing cache from master, [45ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [47ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [48ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [49ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [49ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [125ms] refreshing cache from master, [125ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [127ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [128ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [129ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [130ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [164ms] refreshing cache from master, [164ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [167ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [168ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [169ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [169ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [264ms] refreshing cache from master, [264ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [267ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [268ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [269ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [269ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [1305ms] refreshing cache from master, [1305ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [1307ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=Deferred@597192725(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007fe67c338998@c8a05c8 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1475011386 after retry RPC@1416954551) -> (continuation of Deferred@1924469695 after retry RPC@237623268) -> (continuation of Deferred@1179651198 after retry RPC@518366483) -> (continuation of Deferred@734728762 after retry RPC@414578541) -> (continuation of Deferred@960523390 after retry RPC@251263652), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1e05f97 -> wakeup thread Time-limited test -> (continuation of Deferred@1475011386 after retry RPC@1416954551) -> (continuation of Deferred@1924469695 after retry RPC@237623268) -> (continuation of Deferred@1179651198 after retry RPC@518366483) -> (continuation of Deferred@734728762 after retry RPC@414578541) -> (continuation of Deferred@960523390 after retry RPC@251263652))) after lookup
02:07:25.220 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)] connecting to peer
02:07:25.221 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979
02:07:25.221 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)] cleaning up while in state CONNECTING due to: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979
02:07:25.221 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979) for tablet 5637f0c18ff54bdd81b35dab09fada4d: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979
02:07:25.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:25.228895 23722 raft_consensus.cc:493] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 34e2861a29784992b9d1a9a5876087a6)
02:07:25.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:25.229084 23722 raft_consensus.cc:515] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:25.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:25.229732 23722 leader_election.cc:290] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 34e2861a29784992b9d1a9a5876087a6 (127.21.14.66:35979), 790137564e884f989bbd5c36475181af (127.21.14.65:38711)
02:07:25.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:25.232592 23409 leader_election.cc:336] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 34e2861a29784992b9d1a9a5876087a6 (127.21.14.66:35979): Network error: Client connection negotiation failed: client connection to 127.21.14.66:35979: connect: Connection refused (error 111)
02:07:25.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:25.237241 23657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5637f0c18ff54bdd81b35dab09fada4d" candidate_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "790137564e884f989bbd5c36475181af" is_pre_election: true
02:07:25.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:25.237567 23657 raft_consensus.cc:2468] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a735d9d94bb4c2e8d28cad5e18d6785 in term 8.
02:07:25.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:25.238368 23410 leader_election.cc:304] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1a735d9d94bb4c2e8d28cad5e18d6785, 790137564e884f989bbd5c36475181af; no voters: 34e2861a29784992b9d1a9a5876087a6
02:07:25.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:25.238626 23722 raft_consensus.cc:2804] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [term 9 FOLLOWER]: Leader pre-election won for term 10
02:07:25.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:25.238734 23722 raft_consensus.cc:493] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 34e2861a29784992b9d1a9a5876087a6)
02:07:25.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:25.238842 23722 raft_consensus.cc:3060] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [term 9 FOLLOWER]: Advancing to term 10
02:07:25.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:25.240914 23722 raft_consensus.cc:515] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:25.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:25.241498 23722 leader_election.cc:290] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [CANDIDATE]: Term 10 election: Requested vote from peers 34e2861a29784992b9d1a9a5876087a6 (127.21.14.66:35979), 790137564e884f989bbd5c36475181af (127.21.14.65:38711)
02:07:25.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:25.241946 23657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5637f0c18ff54bdd81b35dab09fada4d" candidate_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "790137564e884f989bbd5c36475181af"
02:07:25.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:25.242136 23657 raft_consensus.cc:3060] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 8 FOLLOWER]: Advancing to term 10
02:07:25.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:25.242756 23409 leader_election.cc:336] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 34e2861a29784992b9d1a9a5876087a6 (127.21.14.66:35979): Network error: Client connection negotiation failed: client connection to 127.21.14.66:35979: connect: Connection refused (error 111)
02:07:25.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:25.245761 23657 raft_consensus.cc:2468] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a735d9d94bb4c2e8d28cad5e18d6785 in term 10.
02:07:25.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:25.246151 23410 leader_election.cc:304] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1a735d9d94bb4c2e8d28cad5e18d6785, 790137564e884f989bbd5c36475181af; no voters: 34e2861a29784992b9d1a9a5876087a6
02:07:25.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:25.246402 23722 raft_consensus.cc:2804] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [term 10 FOLLOWER]: Leader election won for term 10
02:07:25.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:25.246781 23722 raft_consensus.cc:697] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [term 10 LEADER]: Becoming Leader. State: Replica: 1a735d9d94bb4c2e8d28cad5e18d6785, State: Running, Role: LEADER
02:07:25.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:25.247222 23722 consensus_queue.cc:237] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 9.36, Last appended by leader: 36, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:25.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:25.250608 21601 catalog_manager.cc:5697] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 reported cstate change: term changed from 9 to 10, leader changed from 34e2861a29784992b9d1a9a5876087a6 (127.21.14.66) to 1a735d9d94bb4c2e8d28cad5e18d6785 (127.21.14.67). New cstate: current_term: 10 leader_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } health_report { overall_health: UNKNOWN } } }
02:07:25.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:25.725205 23657 raft_consensus.cc:1275] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 10 FOLLOWER]: Refusing update from remote peer 1a735d9d94bb4c2e8d28cad5e18d6785: Log matching property violated. Preceding OpId in replica: term: 8 index: 32. Preceding OpId from leader: term: 10 index: 37. (index mismatch)
02:07:25.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:25.726079 23722 consensus_queue.cc:1048] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [LEADER]: Connected to new peer: Peer: permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 32, Time since last communication: 0.000s
02:07:25.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:25.735430 23741 mvcc.cc:204] Tried to move back new op lower bound from 7305119511813890048 to 7305119508395233280. Current Snapshot: MvccSnapshot[applied={T|T < 7305119511813890048}]
02:07:25.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:25.741761 23409 consensus_peers.cc:597] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 -> Peer 34e2861a29784992b9d1a9a5876087a6 (127.21.14.66:35979): Couldn't send request to peer 34e2861a29784992b9d1a9a5876087a6. Status: Network error: Client connection negotiation failed: client connection to 127.21.14.66:35979: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
02:07:27.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:27.443796 23409 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.14.66:35979: connect: Connection refused (error 111) [suppressed 5 similar messages]
02:07:28.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:28.564105 23409 consensus_peers.cc:597] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 -> Peer 34e2861a29784992b9d1a9a5876087a6 (127.21.14.66:35979): Couldn't send request to peer 34e2861a29784992b9d1a9a5876087a6. Status: Network error: Client connection negotiation failed: client connection to 127.21.14.66:35979: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
02:07:28.758 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:28.759 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)[L],34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979),790137564e884f989bbd5c36475181af(127.21.14.65:38711)]}]
02:07:28.759 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5637f0c18ff54bdd81b35dab09fada4d, attempt=15, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=4848), Traces: [0ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [0ms] delaying RPC due to: Illegal state: connection is terminated, [4ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [6ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [6ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [24ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [27ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [28ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [29ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [29ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [45ms] refreshing cache from master, [45ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [47ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [48ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [49ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [49ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [125ms] refreshing cache from master, [125ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [127ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [128ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [129ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [130ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [164ms] refreshing cache from master, [164ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [167ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [168ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [169ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [169ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [264ms] refreshing cache from master, [264ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [267ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [268ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [269ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [269ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [1305ms] refreshing cache from master, [1305ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [1307ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [1309ms] sending RPC to server 34e2861a29784992b9d1a9a5876087a6, [1310ms] received response from server 34e2861a29784992b9d1a9a5876087a6: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [1310ms] delaying RPC due to: Network error: Failed to connect to peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979): Connection refused: /127.21.14.66:35979, [4845ms] refreshing cache from master, [4845ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [4847ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=Deferred@597192725(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007fe67c338998@c8a05c8 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1475011386 after retry RPC@1416954551) -> (continuation of Deferred@1924469695 after retry RPC@237623268) -> (continuation of Deferred@1179651198 after retry RPC@518366483) -> (continuation of Deferred@734728762 after retry RPC@414578541) -> (continuation of Deferred@960523390 after retry RPC@251263652) -> (continuation of Deferred@59928027 after retry RPC@1124047831), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1e05f97 -> wakeup thread Time-limited test -> (continuation of Deferred@1475011386 after retry RPC@1416954551) -> (continuation of Deferred@1924469695 after retry RPC@237623268) -> (continuation of Deferred@1179651198 after retry RPC@518366483) -> (continuation of Deferred@734728762 after retry RPC@414578541) -> (continuation of Deferred@960523390 after retry RPC@251263652) -> (continuation of Deferred@59928027 after retry RPC@1124047831))) after lookup
02:07:28.760 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)] connecting to peer
02:07:28.760 [DEBUG - kudu-nio-4] (Connection.java:757) [peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)] Successfully connected to peer
02:07:28.774 [DEBUG - kudu-nio-4] (Negotiator.java:918) Authenticated connection [id: 0xea53d582, L:/127.0.0.1:43288 - R:/127.21.14.67:42423] using TOKEN/null
02:07:28.793 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.21.14.66:35979
02:07:28.793 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

02:07:28.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:28.794214 21561 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu
02:07:28.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu
02:07:28.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-1/wal
02:07:28.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-1/data
02:07:28.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:07:28.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:07:28.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:07:28.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:07:28.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:07:28.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:07:28.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-1/logs
02:07:28.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-1/data/info.pb
02:07:28.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:07:28.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:07:28.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:07:28.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:07:28.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:07:28.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:07:28.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:07:28.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:07:28.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:07:28.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:07:28.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.14.66:35979
02:07:28.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.21.14.66
02:07:28.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.21.14.126:34291,127.21.14.125:33787,127.21.14.124:36603
02:07:28.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=43569
02:07:28.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.14.66
02:07:28.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.14.84:44189
02:07:28.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:07:28.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:07:28.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:07:29.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:29.103129 23771 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:07:29.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:29.103428 23771 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:07:29.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:29.103544 23771 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:07:29.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:29.112406 23771 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:07:29.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:29.112632 23771 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.14.66
02:07:29.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:29.122426 23771 tablet_server_runner.cc:78] Tablet server non-default flags:
02:07:29.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:07:29.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.14.84:44189
02:07:29.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:07:29.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:07:29.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-1/data
02:07:29.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-1/wal
02:07:29.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:07:29.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:07:29.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.14.66:35979
02:07:29.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:07:29.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:07:29.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:07:29.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-1/data/info.pb
02:07:29.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.14.66
02:07:29.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=43569
02:07:29.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.21.14.126:34291,127.21.14.125:33787,127.21.14.124:36603
02:07:29.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:07:29.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:07:29.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:07:29.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:07:29.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:07:29.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.21.14.66
02:07:29.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-1/logs
02:07:29.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:07:29.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:07:29.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:07:29.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:07:29.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:07:29.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
02:07:29.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:07:29.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:43:30 UTC on bdcb31816ec0
02:07:29.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12766
02:07:29.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
02:07:29.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:29.124276 23771 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:07:29.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:29.126035 23771 file_cache.cc:504] Constructed file cache file cache with capacity 419430
02:07:29.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:29.134828 23778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:29.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:29.135138 23779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:29.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:29.136044 23781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:29.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:29.136592 23771 server_base.cc:1061] running on GCE node
02:07:29.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:29.137099 23771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:07:29.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:29.138186 23771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:07:29.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:29.139434 23771 hybrid_clock.cc:648] HybridClock initialized: now 1783476449139373 us; error 60 us; skew 500 ppm
02:07:29.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:29.139792 23771 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:07:29.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:29.142302 23771 webserver.cc:533] Webserver started at http://127.21.14.66:43569/ using document root <none> and password file <none>
02:07:29.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:29.143219 23771 fs_manager.cc:362] Metadata directory not provided
02:07:29.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:29.143345 23771 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:07:29.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:29.150022 23771 fs_manager.cc:714] Time spent opening directory manager: real 0.004s	user 0.001s	sys 0.003s
02:07:29.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:29.153260 23787 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:07:29.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:29.154547 23771 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
02:07:29.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:29.154685 23771 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-1/data,/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-1/wal
02:07:29.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "34e2861a29784992b9d1a9a5876087a6"
02:07:29.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:06:59 on dist-test-slave-545r"
02:07:29.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:29.155310 23771 fs_report.cc:389] FS layout report
02:07:29.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:07:29.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-1/wal
02:07:29.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-1/wal
02:07:29.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-1/data/data
02:07:29.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:07:29.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:07:29.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:07:29.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:07:29.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:07:29.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:07:29.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:07:29.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:07:29.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:07:29.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:07:29.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:07:29.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:29.175693 23771 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:07:29.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:29.176451 23771 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:07:29.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:29.176801 23771 kserver.cc:163] Server-wide thread pool size limit: 3276
02:07:29.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:29.177668 23771 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:07:29.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:29.179575 23794 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
02:07:29.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:29.184989 23771 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
02:07:29.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:29.185077 23771 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.006s	user 0.001s	sys 0.000s
02:07:29.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:29.185166 23771 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
02:07:29.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:29.187953 23771 ts_tablet_manager.cc:616] Registered 1 tablets
02:07:29.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:29.188052 23771 ts_tablet_manager.cc:595] Time spent register tablets: real 0.003s	user 0.000s	sys 0.000s
02:07:29.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:29.188643 23794 tablet_bootstrap.cc:492] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6: Bootstrap starting.
02:07:29.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:29.214371 23794 log.cc:826] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6: Log is configured to *not* fsync() on all Append() calls
02:07:29.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:29.237054 23771 rpc_server.cc:307] RPC server started. Bound to: 127.21.14.66:35979
02:07:29.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:29.237073 23901 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.14.66:35979 every 8 connection(s)
02:07:29.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:29.238816 23771 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-1/data/info.pb
02:07:29.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:29.247644 21561 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu as pid 23771
02:07:29.248 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:07:29.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:29.257651 23794 tablet_bootstrap.cc:492] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6: Bootstrap replayed 1/1 log segments. Stats: ops{read=36 overwritten=0 applied=36 ignored=0} inserts{seen=27 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
02:07:29.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:29.258529 23794 tablet_bootstrap.cc:492] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6: Bootstrap complete.
02:07:29.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:29.259958 23794 ts_tablet_manager.cc:1403] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6: Time spent bootstrapping tablet: real 0.072s	user 0.060s	sys 0.008s
02:07:29.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:29.266758 23794 raft_consensus.cc:359] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [term 9 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:29.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:29.267753 23794 raft_consensus.cc:740] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [term 9 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34e2861a29784992b9d1a9a5876087a6, State: Initialized, Role: FOLLOWER
02:07:29.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:29.268532 23794 consensus_queue.cc:260] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 36, Last appended: 9.36, Last appended by leader: 36, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:29.274 [DEBUG - kudu-nio-4] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 30 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283, tablet=5637f0c18ff54bdd81b35dab09fada4d, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:07:29.275 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.21.14.67:42423
02:07:29.275 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

02:07:29.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:29.275929 21561 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu pid 23394
02:07:29.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:29.294310 23794 ts_tablet_manager.cc:1434] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6: Time spent starting tablet: real 0.034s	user 0.013s	sys 0.004s
02:07:29.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:29.302235 23518 generic_service.cc:196] Checking for leaks (request via RPC)
02:07:29.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:29.303185 23904 heartbeater.cc:344] Connected to a master server at 127.21.14.125:33787
02:07:29.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:29.303608 23904 heartbeater.cc:461] Registering TS with master...
02:07:29.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:29.304730 23904 heartbeater.cc:507] Master 127.21.14.125:33787 requested a full tablet report, sending...
02:07:29.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:29.307935 21673 ts_manager.cc:194] Re-registered known tserver with Master: 34e2861a29784992b9d1a9a5876087a6 (127.21.14.66:35979)
02:07:29.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:29.309314 23903 heartbeater.cc:344] Connected to a master server at 127.21.14.126:34291
02:07:29.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:29.309439 23903 heartbeater.cc:461] Registering TS with master...
02:07:29.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:29.309924 23903 heartbeater.cc:507] Master 127.21.14.126:34291 requested a full tablet report, sending...
02:07:29.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:29.311347 21601 ts_manager.cc:194] Re-registered known tserver with Master: 34e2861a29784992b9d1a9a5876087a6 (127.21.14.66:35979)
02:07:29.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:29.311575 23902 heartbeater.cc:344] Connected to a master server at 127.21.14.124:36603
02:07:29.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:29.311685 23902 heartbeater.cc:461] Registering TS with master...
02:07:29.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:29.312073 23902 heartbeater.cc:507] Master 127.21.14.124:36603 requested a full tablet report, sending...
02:07:29.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:29.312884 21601 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.21.14.66:53913
02:07:29.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:29.313283 21745 ts_manager.cc:194] Re-registered known tserver with Master: 34e2861a29784992b9d1a9a5876087a6 (127.21.14.66:35979)
02:07:29.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:29.315442 23903 heartbeater.cc:499] Master 127.21.14.126:34291 was elected leader, sending a full tablet report...
02:07:29.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:29.407876 21561 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu with pid 23394
02:07:29.422 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:07:29.423 [DEBUG - kudu-nio-4] (Connection.java:239) [peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)] handling channelInactive
02:07:29.423 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)] cleaning up while in state READY due to: connection closed
02:07:29.436 [DEBUG - pool-2-thread-1] (Connection.java:748) [peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)] connecting to peer
02:07:29.437 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:29.437 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:29.437 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423) for tablet 5637f0c18ff54bdd81b35dab09fada4d: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:29.458 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:29.459 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)[L],34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979),790137564e884f989bbd5c36475181af(127.21.14.65:38711)]}]
02:07:29.459 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5637f0c18ff54bdd81b35dab09fada4d, attempt=3, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=36), Traces: [0ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [0ms] delaying RPC due to: Illegal state: connection is terminated, [13ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [14ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [14ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [35ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=Deferred@1110090284(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007fe67c338998@7cb5a89d -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5b99d5d5 -> wakeup thread Time-limited test)) after lookup
02:07:29.459 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)] connecting to peer
02:07:29.460 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:29.460 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:29.461 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423) for tablet 5637f0c18ff54bdd81b35dab09fada4d: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:29.478 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:29.478 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)[L],34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979),790137564e884f989bbd5c36475181af(127.21.14.65:38711)]}]
02:07:29.478 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5637f0c18ff54bdd81b35dab09fada4d, attempt=5, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=55), Traces: [0ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [0ms] delaying RPC due to: Illegal state: connection is terminated, [13ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [14ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [14ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [35ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [36ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [37ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [38ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [52ms] refreshing cache from master, [52ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [54ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=Deferred@1110090284(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007fe67c338998@7cb5a89d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2017393627 after retry RPC@1838984820), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5b99d5d5 -> wakeup thread Time-limited test -> (continuation of Deferred@2017393627 after retry RPC@1838984820))) after lookup
02:07:29.478 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)] connecting to peer
02:07:29.479 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:29.479 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:29.479 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423) for tablet 5637f0c18ff54bdd81b35dab09fada4d: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:29.538 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:29.539 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)[L],34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979),790137564e884f989bbd5c36475181af(127.21.14.65:38711)]}]
02:07:29.539 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5637f0c18ff54bdd81b35dab09fada4d, attempt=7, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=116), Traces: [0ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [0ms] delaying RPC due to: Illegal state: connection is terminated, [13ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [14ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [14ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [35ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [36ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [37ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [38ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [52ms] refreshing cache from master, [52ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [54ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [55ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [56ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [56ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [113ms] refreshing cache from master, [113ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [115ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=Deferred@1110090284(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007fe67c338998@7cb5a89d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2017393627 after retry RPC@1838984820) -> (continuation of Deferred@308156494 after retry RPC@440007551), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5b99d5d5 -> wakeup thread Time-limited test -> (continuation of Deferred@2017393627 after retry RPC@1838984820) -> (continuation of Deferred@308156494 after retry RPC@440007551))) after lookup
02:07:29.539 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)] connecting to peer
02:07:29.540 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:29.540 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:29.540 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423) for tablet 5637f0c18ff54bdd81b35dab09fada4d: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:29.578 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:29.578 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)[L],34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979),790137564e884f989bbd5c36475181af(127.21.14.65:38711)]}]
02:07:29.578 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5637f0c18ff54bdd81b35dab09fada4d, attempt=9, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=155), Traces: [0ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [0ms] delaying RPC due to: Illegal state: connection is terminated, [13ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [14ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [14ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [35ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [36ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [37ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [38ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [52ms] refreshing cache from master, [52ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [54ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [55ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [56ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [56ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [113ms] refreshing cache from master, [113ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [115ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [116ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [117ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [117ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [152ms] refreshing cache from master, [153ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [155ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=Deferred@1110090284(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007fe67c338998@7cb5a89d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2017393627 after retry RPC@1838984820) -> (continuation of Deferred@308156494 after retry RPC@440007551) -> (continuation of Deferred@1639402349 after retry RPC@338688097), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5b99d5d5 -> wakeup thread Time-limited test -> (continuation of Deferred@2017393627 after retry RPC@1838984820) -> (continuation of Deferred@308156494 after retry RPC@440007551) -> (continuation of Deferred@1639402349 after retry RPC@338688097))) after lookup
02:07:29.579 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)] connecting to peer
02:07:29.580 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:29.580 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:29.580 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423) for tablet 5637f0c18ff54bdd81b35dab09fada4d: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:29.799 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:29.799 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)[L],34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979),790137564e884f989bbd5c36475181af(127.21.14.65:38711)]}]
02:07:29.799 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5637f0c18ff54bdd81b35dab09fada4d, attempt=11, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=376), Traces: [0ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [0ms] delaying RPC due to: Illegal state: connection is terminated, [13ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [14ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [14ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [35ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [36ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [37ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [38ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [52ms] refreshing cache from master, [52ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [54ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [55ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [56ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [56ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [113ms] refreshing cache from master, [113ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [115ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [116ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [117ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [117ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [152ms] refreshing cache from master, [153ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [155ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [156ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [157ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [157ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [373ms] refreshing cache from master, [373ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [375ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=Deferred@1110090284(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007fe67c338998@7cb5a89d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2017393627 after retry RPC@1838984820) -> (continuation of Deferred@308156494 after retry RPC@440007551) -> (continuation of Deferred@1639402349 after retry RPC@338688097) -> (continuation of Deferred@621368947 after retry RPC@1317953091), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5b99d5d5 -> wakeup thread Time-limited test -> (continuation of Deferred@2017393627 after retry RPC@1838984820) -> (continuation of Deferred@308156494 after retry RPC@440007551) -> (continuation of Deferred@1639402349 after retry RPC@338688097) -> (continuation of Deferred@621368947 after retry RPC@1317953091))) after lookup
02:07:29.800 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)] connecting to peer
02:07:29.801 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:29.801 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:29.801 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423) for tablet 5637f0c18ff54bdd81b35dab09fada4d: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423
02:07:30.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:30.747008 23917 raft_consensus.cc:493] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
02:07:30.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:30.747234 23917 raft_consensus.cc:515] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:30.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:30.748739 23917 leader_election.cc:290] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1a735d9d94bb4c2e8d28cad5e18d6785 (127.21.14.67:42423), 790137564e884f989bbd5c36475181af (127.21.14.65:38711)
02:07:30.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:30.750373 23789 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.14.67:42423: connect: Connection refused (error 111)
02:07:30.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:30.752710 23789 leader_election.cc:336] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 1a735d9d94bb4c2e8d28cad5e18d6785 (127.21.14.67:42423): Network error: Client connection negotiation failed: client connection to 127.21.14.67:42423: connect: Connection refused (error 111)
02:07:30.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:30.756888 23657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5637f0c18ff54bdd81b35dab09fada4d" candidate_uuid: "34e2861a29784992b9d1a9a5876087a6" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "790137564e884f989bbd5c36475181af" is_pre_election: true
02:07:30.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:30.757573 23789 leader_election.cc:304] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 34e2861a29784992b9d1a9a5876087a6; no voters: 1a735d9d94bb4c2e8d28cad5e18d6785, 790137564e884f989bbd5c36475181af
02:07:30.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:30.757925 23917 raft_consensus.cc:2749] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
02:07:30.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:30.799768 23922 raft_consensus.cc:493] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 1a735d9d94bb4c2e8d28cad5e18d6785)
02:07:30.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:30.799937 23922 raft_consensus.cc:515] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:30.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:30.800525 23922 leader_election.cc:290] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1a735d9d94bb4c2e8d28cad5e18d6785 (127.21.14.67:42423), 34e2861a29784992b9d1a9a5876087a6 (127.21.14.66:35979)
02:07:30.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:30.801826 23590 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.14.67:42423: connect: Connection refused (error 111)
02:07:30.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:30.803282 23590 leader_election.cc:336] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 1a735d9d94bb4c2e8d28cad5e18d6785 (127.21.14.67:42423): Network error: Client connection negotiation failed: client connection to 127.21.14.67:42423: connect: Connection refused (error 111)
02:07:30.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:30.808524 23857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5637f0c18ff54bdd81b35dab09fada4d" candidate_uuid: "790137564e884f989bbd5c36475181af" candidate_term: 11 candidate_status { last_received { term: 10 index: 40 } } ignore_live_leader: false dest_uuid: "34e2861a29784992b9d1a9a5876087a6" is_pre_election: true
02:07:30.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:30.808929 23857 raft_consensus.cc:2468] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 790137564e884f989bbd5c36475181af in term 9.
02:07:30.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:30.809695 23589 leader_election.cc:304] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 34e2861a29784992b9d1a9a5876087a6, 790137564e884f989bbd5c36475181af; no voters: 1a735d9d94bb4c2e8d28cad5e18d6785
02:07:30.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:30.809962 23922 raft_consensus.cc:2804] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 10 FOLLOWER]: Leader pre-election won for term 11
02:07:30.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:30.810067 23922 raft_consensus.cc:493] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 10 FOLLOWER]: Starting leader election (detected failure of leader 1a735d9d94bb4c2e8d28cad5e18d6785)
02:07:30.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:30.810187 23922 raft_consensus.cc:3060] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 10 FOLLOWER]: Advancing to term 11
02:07:30.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:30.812139 23922 raft_consensus.cc:515] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:30.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:30.812753 23922 leader_election.cc:290] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [CANDIDATE]: Term 11 election: Requested vote from peers 1a735d9d94bb4c2e8d28cad5e18d6785 (127.21.14.67:42423), 34e2861a29784992b9d1a9a5876087a6 (127.21.14.66:35979)
02:07:30.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:30.813213 23857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5637f0c18ff54bdd81b35dab09fada4d" candidate_uuid: "790137564e884f989bbd5c36475181af" candidate_term: 11 candidate_status { last_received { term: 10 index: 40 } } ignore_live_leader: false dest_uuid: "34e2861a29784992b9d1a9a5876087a6"
02:07:30.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:30.813388 23857 raft_consensus.cc:3060] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [term 9 FOLLOWER]: Advancing to term 11
02:07:30.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:30.814075 23590 leader_election.cc:336] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 1a735d9d94bb4c2e8d28cad5e18d6785 (127.21.14.67:42423): Network error: Client connection negotiation failed: client connection to 127.21.14.67:42423: connect: Connection refused (error 111)
02:07:30.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:30.818606 23857 raft_consensus.cc:2468] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 790137564e884f989bbd5c36475181af in term 11.
02:07:30.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:30.819036 23589 leader_election.cc:304] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 34e2861a29784992b9d1a9a5876087a6, 790137564e884f989bbd5c36475181af; no voters: 1a735d9d94bb4c2e8d28cad5e18d6785
02:07:30.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:30.819281 23922 raft_consensus.cc:2804] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 11 FOLLOWER]: Leader election won for term 11
02:07:30.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:30.819737 23922 raft_consensus.cc:697] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [term 11 LEADER]: Becoming Leader. State: Replica: 790137564e884f989bbd5c36475181af, State: Running, Role: LEADER
02:07:30.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:30.820245 23922 consensus_queue.cc:237] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 10.40, Last appended by leader: 40, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:30.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:30.823892 21601 catalog_manager.cc:5697] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af reported cstate change: term changed from 10 to 11, leader changed from 1a735d9d94bb4c2e8d28cad5e18d6785 (127.21.14.67) to 790137564e884f989bbd5c36475181af (127.21.14.65). New cstate: current_term: 11 leader_uuid: "790137564e884f989bbd5c36475181af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } health_report { overall_health: HEALTHY } } }
02:07:31.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:31.221617 23857 raft_consensus.cc:1275] T 5637f0c18ff54bdd81b35dab09fada4d P 34e2861a29784992b9d1a9a5876087a6 [term 11 FOLLOWER]: Refusing update from remote peer 790137564e884f989bbd5c36475181af: Log matching property violated. Preceding OpId in replica: term: 9 index: 36. Preceding OpId from leader: term: 11 index: 41. (index mismatch)
02:07:31.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:31.222407 23922 consensus_queue.cc:1048] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af [LEADER]: Connected to new peer: Peer: permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 36, Time since last communication: 0.000s
02:07:31.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:31.236096 23590 consensus_peers.cc:597] T 5637f0c18ff54bdd81b35dab09fada4d P 790137564e884f989bbd5c36475181af -> Peer 1a735d9d94bb4c2e8d28cad5e18d6785 (127.21.14.67:42423): Couldn't send request to peer 1a735d9d94bb4c2e8d28cad5e18d6785. Status: Network error: Client connection negotiation failed: client connection to 127.21.14.67:42423: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
02:07:31.939 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 5637f0c18ff54bdd81b35dab09fada4d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283' with partition [<start>, <end>)
02:07:31.939 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5637f0c18ff54bdd81b35dab09fada4d@[1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423),34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979),790137564e884f989bbd5c36475181af(127.21.14.65:38711)[L]]}]
02:07:31.940 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5637f0c18ff54bdd81b35dab09fada4d, attempt=13, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=2516), Traces: [0ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [0ms] delaying RPC due to: Illegal state: connection is terminated, [13ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [14ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [14ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [35ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [36ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [37ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [38ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [52ms] refreshing cache from master, [52ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [54ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [55ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [56ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [56ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [113ms] refreshing cache from master, [113ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [115ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [116ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [117ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [117ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [152ms] refreshing cache from master, [153ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [155ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [156ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [157ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [157ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [373ms] refreshing cache from master, [373ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [375ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, [377ms] sending RPC to server 1a735d9d94bb4c2e8d28cad5e18d6785, [378ms] received response from server 1a735d9d94bb4c2e8d28cad5e18d6785: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [378ms] delaying RPC due to: Network error: Failed to connect to peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423): Connection refused: /127.21.14.67:42423, [2513ms] refreshing cache from master, [2513ms] Sub RPC GetTableLocations: sending RPC to server master-127.21.14.126:34291, [2516ms] Sub RPC GetTableLocations: received response from server master-127.21.14.126:34291: OK, deferred=Deferred@1110090284(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$234/0x00007fe67c338998@7cb5a89d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2017393627 after retry RPC@1838984820) -> (continuation of Deferred@308156494 after retry RPC@440007551) -> (continuation of Deferred@1639402349 after retry RPC@338688097) -> (continuation of Deferred@621368947 after retry RPC@1317953091) -> (continuation of Deferred@529262019 after retry RPC@592174135), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5b99d5d5 -> wakeup thread Time-limited test -> (continuation of Deferred@2017393627 after retry RPC@1838984820) -> (continuation of Deferred@308156494 after retry RPC@440007551) -> (continuation of Deferred@1639402349 after retry RPC@338688097) -> (continuation of Deferred@621368947 after retry RPC@1317953091) -> (continuation of Deferred@529262019 after retry RPC@592174135))) after lookup
02:07:31.940 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711)] connecting to peer
02:07:31.941 [DEBUG - kudu-nio-3] (Connection.java:757) [peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711)] Successfully connected to peer
02:07:31.961 [DEBUG - kudu-nio-3] (Negotiator.java:918) Authenticated connection [id: 0xefa99ee5, L:/127.0.0.1:41266 - R:/127.21.14.65:38711] using TOKEN/null
02:07:31.981 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.21.14.67:42423
02:07:31.981 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

02:07:31.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:31.982447 21561 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu
02:07:31.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu
02:07:31.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/wal
02:07:31.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/data
02:07:31.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:07:31.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:07:31.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:07:31.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:07:31.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:07:31.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:07:31.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/logs
02:07:31.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/data/info.pb
02:07:31.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:07:31.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:07:31.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:07:31.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:07:31.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:07:31.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:07:31.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:07:31.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:07:31.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:07:31.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:07:31.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.14.67:42423
02:07:31.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.21.14.67
02:07:31.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.21.14.126:34291,127.21.14.125:33787,127.21.14.124:36603
02:07:31.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=37049
02:07:31.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.14.67
02:07:31.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.14.84:44189
02:07:31.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:07:31.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:07:31.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:07:32.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:32.284252 23955 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:07:32.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:32.284606 23955 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:07:32.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:32.284700 23955 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:07:32.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:32.292819 23955 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:07:32.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:32.292958 23955 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.14.67
02:07:32.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.302407 23955 tablet_server_runner.cc:78] Tablet server non-default flags:
02:07:32.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:07:32.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.14.84:44189
02:07:32.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:07:32.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:07:32.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/data
02:07:32.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/wal
02:07:32.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:07:32.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:07:32.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.14.67:42423
02:07:32.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:07:32.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:07:32.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:07:32.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/data/info.pb
02:07:32.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.14.67
02:07:32.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=37049
02:07:32.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.21.14.126:34291,127.21.14.125:33787,127.21.14.124:36603
02:07:32.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:07:32.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:07:32.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:07:32.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:07:32.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:07:32.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.21.14.67
02:07:32.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/logs
02:07:32.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:07:32.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:07:32.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:07:32.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:07:32.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:07:32.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
02:07:32.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:07:32.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:43:30 UTC on bdcb31816ec0
02:07:32.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12766
02:07:32.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
02:07:32.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.304031 23955 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:07:32.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.306308 23955 file_cache.cc:504] Constructed file cache file cache with capacity 419430
02:07:32.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:32.319214 23962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:32.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:32.319855 23963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:32.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:32.320261 23965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:32.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.320670 23955 server_base.cc:1061] running on GCE node
02:07:32.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.321189 23955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:07:32.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.321980 23955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:07:32.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.323169 23955 hybrid_clock.cc:648] HybridClock initialized: now 1783476452323123 us; error 33 us; skew 500 ppm
02:07:32.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.323474 23955 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:07:32.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.325506 23955 webserver.cc:533] Webserver started at http://127.21.14.67:37049/ using document root <none> and password file <none>
02:07:32.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.326193 23955 fs_manager.cc:362] Metadata directory not provided
02:07:32.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.326292 23955 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:07:32.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.331187 23955 fs_manager.cc:714] Time spent opening directory manager: real 0.003s	user 0.001s	sys 0.004s
02:07:32.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.333704 23971 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:07:32.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.334848 23955 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.000s	sys 0.003s
02:07:32.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.334990 23955 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/data,/tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/wal
02:07:32.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "1a735d9d94bb4c2e8d28cad5e18d6785"
02:07:32.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:06:59 on dist-test-slave-545r"
02:07:32.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.335685 23955 fs_report.cc:389] FS layout report
02:07:32.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:07:32.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/wal
02:07:32.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/wal
02:07:32.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/data/data
02:07:32.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:07:32.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:07:32.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:07:32.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:07:32.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:07:32.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:07:32.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:07:32.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:07:32.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:07:32.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:07:32.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:07:32.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.358435 23955 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:07:32.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.359258 23955 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:07:32.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.359558 23955 kserver.cc:163] Server-wide thread pool size limit: 3276
02:07:32.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.360510 23955 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:07:32.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.362545 23978 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
02:07:32.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.367949 23955 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
02:07:32.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.368033 23955 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.006s	user 0.001s	sys 0.000s
02:07:32.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.368158 23955 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
02:07:32.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.371100 23955 ts_tablet_manager.cc:616] Registered 1 tablets
02:07:32.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.371191 23955 ts_tablet_manager.cc:595] Time spent register tablets: real 0.003s	user 0.001s	sys 0.000s
02:07:32.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.371552 23978 tablet_bootstrap.cc:492] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785: Bootstrap starting.
02:07:32.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.398231 23978 log.cc:826] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785: Log is configured to *not* fsync() on all Append() calls
02:07:32.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.405674 23955 rpc_server.cc:307] RPC server started. Bound to: 127.21.14.67:42423
02:07:32.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.405707 24085 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.14.67:42423 every 8 connection(s)
02:07:32.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.407039 23955 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQUnVk7/test-tmp/mini-kudu-cluster8923021907733061309/ts-2/data/info.pb
02:07:32.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.416713 21561 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu as pid 23955
02:07:32.417 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:07:32.434 [DEBUG - kudu-nio-3] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 33 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283, tablet=5637f0c18ff54bdd81b35dab09fada4d, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:07:32.440 [DEBUG - kudu-nio-3] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 33 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783476420283, tablet=5637f0c18ff54bdd81b35dab09fada4d, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:07:32.446 [DEBUG - main] (Connection.java:575) [peer master-127.21.14.124:36603(127.21.14.124:36603)] disconnecting while in state READY
02:07:32.447 [DEBUG - main] (Connection.java:575) [peer master-127.21.14.126:34291(127.21.14.126:34291)] disconnecting while in state READY
02:07:32.447 [DEBUG - main] (Connection.java:575) [peer 34e2861a29784992b9d1a9a5876087a6(127.21.14.66:35979)] disconnecting while in state TERMINATED
02:07:32.447 [DEBUG - main] (Connection.java:575) [peer master-127.21.14.125:33787(127.21.14.125:33787)] disconnecting while in state READY
02:07:32.447 [DEBUG - main] (Connection.java:575) [peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711)] disconnecting while in state READY
02:07:32.447 [DEBUG - main] (Connection.java:575) [peer 1a735d9d94bb4c2e8d28cad5e18d6785(127.21.14.67:42423)] disconnecting while in state TERMINATED
02:07:32.449 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.21.14.126:34291(127.21.14.126:34291)] handling channelInactive
02:07:32.449 [DEBUG - kudu-nio-2] (Connection.java:239) [peer master-127.21.14.125:33787(127.21.14.125:33787)] handling channelInactive
02:07:32.449 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.21.14.126:34291(127.21.14.126:34291)] cleaning up while in state READY due to: connection closed
02:07:32.450 [DEBUG - kudu-nio-3] (Connection.java:239) [peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711)] handling channelInactive
02:07:32.450 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 790137564e884f989bbd5c36475181af(127.21.14.65:38711)] cleaning up while in state READY due to: connection closed
02:07:32.449 [DEBUG - kudu-nio-2] (Connection.java:709) [peer master-127.21.14.125:33787(127.21.14.125:33787)] cleaning up while in state READY due to: connection closed
02:07:32.452 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2704) Releasing all remaining resources
02:07:32.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.452976 23978 tablet_bootstrap.cc:492] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785: Bootstrap replayed 1/1 log segments. Stats: ops{read=40 overwritten=0 applied=40 ignored=0} inserts{seen=30 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
02:07:32.453 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.21.14.124:36603(127.21.14.124:36603)] handling channelInactive
02:07:32.453 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.21.14.124:36603(127.21.14.124:36603)] cleaning up while in state READY due to: connection closed
02:07:32.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.456185 23978 tablet_bootstrap.cc:492] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785: Bootstrap complete.
02:07:32.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.458303 23978 ts_tablet_manager.cc:1403] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785: Time spent bootstrapping tablet: real 0.087s	user 0.050s	sys 0.020s
02:07:32.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.460047 21561 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu pid 23572
02:07:32.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.467192 24086 heartbeater.cc:344] Connected to a master server at 127.21.14.124:36603
02:07:32.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.467595 24086 heartbeater.cc:461] Registering TS with master...
02:07:32.468 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.468701 24086 heartbeater.cc:507] Master 127.21.14.124:36603 requested a full tablet report, sending...
02:07:32.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.469375 23978 raft_consensus.cc:359] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [term 10 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:32.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.470032 23978 raft_consensus.cc:740] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [term 10 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a735d9d94bb4c2e8d28cad5e18d6785, State: Initialized, Role: FOLLOWER
02:07:32.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.470659 23978 consensus_queue.cc:260] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 40, Last appended: 10.40, Last appended by leader: 40, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a735d9d94bb4c2e8d28cad5e18d6785" member_type: VOTER last_known_addr { host: "127.21.14.67" port: 42423 } } peers { permanent_uuid: "34e2861a29784992b9d1a9a5876087a6" member_type: VOTER last_known_addr { host: "127.21.14.66" port: 35979 } } peers { permanent_uuid: "790137564e884f989bbd5c36475181af" member_type: VOTER last_known_addr { host: "127.21.14.65" port: 38711 } }
02:07:32.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.471771 21745 ts_manager.cc:194] Re-registered known tserver with Master: 1a735d9d94bb4c2e8d28cad5e18d6785 (127.21.14.67:42423)
02:07:32.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.472376 23978 ts_tablet_manager.cc:1434] T 5637f0c18ff54bdd81b35dab09fada4d P 1a735d9d94bb4c2e8d28cad5e18d6785: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.001s
02:07:32.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.480299 24087 heartbeater.cc:344] Connected to a master server at 127.21.14.126:34291
02:07:32.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.480476 24087 heartbeater.cc:461] Registering TS with master...
02:07:32.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.480949 24087 heartbeater.cc:507] Master 127.21.14.126:34291 requested a full tablet report, sending...
02:07:32.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.482381 21601 ts_manager.cc:194] Re-registered known tserver with Master: 1a735d9d94bb4c2e8d28cad5e18d6785 (127.21.14.67:42423)
02:07:32.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.483556 24088 heartbeater.cc:344] Connected to a master server at 127.21.14.125:33787
02:07:32.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.483673 24088 heartbeater.cc:461] Registering TS with master...
02:07:32.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.483795 21601 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.21.14.67:36667
02:07:32.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.484139 24088 heartbeater.cc:507] Master 127.21.14.125:33787 requested a full tablet report, sending...
02:07:32.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.484705 23698 generic_service.cc:196] Checking for leaks (request via RPC)
02:07:32.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.485551 21674 ts_manager.cc:194] Re-registered known tserver with Master: 1a735d9d94bb4c2e8d28cad5e18d6785 (127.21.14.67:42423)
02:07:32.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.486485 24087 heartbeater.cc:499] Master 127.21.14.126:34291 was elected leader, sending a full tablet report...
02:07:32.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.582324 21561 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu with pid 23572
02:07:32.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.596230 21561 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu pid 23771
02:07:32.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.602249 23897 generic_service.cc:196] Checking for leaks (request via RPC)
02:07:32.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.690197 21561 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu with pid 23771
02:07:32.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.703390 21561 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu pid 23955
02:07:32.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.711294 24081 generic_service.cc:196] Checking for leaks (request via RPC)
02:07:32.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.797732 21561 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu with pid 23955
02:07:32.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.810802 21561 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu pid 21572
02:07:32.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.811954 21633 generic_service.cc:196] Checking for leaks (request via RPC)
02:07:32.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.942139 21561 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu with pid 21572
02:07:32.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:07:32.959293 21730 connection.cc:573] server connection from 127.0.0.1:38010 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
02:07:32.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.959937 21561 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu pid 21640
02:07:32.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:32.961093 21704 generic_service.cc:196] Checking for leaks (request via RPC)
02:07:33.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:33.084142 21561 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu with pid 21640
02:07:33.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:33.100840 21561 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu pid 21711
02:07:33.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:33.101953 21775 generic_service.cc:196] Checking for leaks (request via RPC)
02:07:33.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:07:33.221285 21561 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskQUnVk7/build/asan/bin/kudu with pid 21711
02:07:33.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-07-08T02:07:33Z chronyd exiting

Time: 37.643

OK (1 test)

