JUnit version 4.13.2
.02:03:34.131 [DEBUG - main] (RandomUtils.java:49) Using random seed: 1783476214124
02:03:34.181 [INFO - main] (RetryRule.java:138) Creating RetryStatement with result reporter and retry count of 0 (no retries)
02:03:34.210 [INFO - main] (KuduTestHarness.java:155) Creating a new MiniKuduCluster...
02:03:34.220 [INFO - main] (TempDirUtils.java:71) Using the temp directory defined by TEST_TMPDIR: /tmp/dist-test-taskRwVx6v/test-tmp
02:03:34.244 [INFO - main] (KuduBinaryLocator.java:83) Using Kudu binary directory specified by system property 'kuduBinDir': ../../../build/asan/bin
02:03:34.247 [INFO - main] (MiniKuduCluster.java:217) Starting process: [/tmp/dist-test-taskRwVx6v/java/build/dist-test/../../../build/asan/bin/kudu, test, mini_cluster, --serialization=pb]
02:03:34.471 [DEBUG - main] (MiniKuduCluster.java:185) Request: create_cluster {
  num_masters: 3
  num_tservers: 3
  enable_kerberos: false
  cluster_root: "/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638"
  hms_mode: NONE
  mini_kdc_options {
  }
  principal: "kudu"
  mini_oidc_options {
  }
}

02:03:34.655 [DEBUG - main] (MiniKuduCluster.java:195) Response: 
02:03:34.657 [DEBUG - main] (MiniKuduCluster.java:185) Request: start_cluster {
}

02:03:34.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:34.661106 14284 mini_oidc.cc:150] Starting JWKS server
02:03:34.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:34.664521 14284 webserver.cc:534] Webserver started at http://127.0.0.1:35485/ using document root <none> and password file <none>
02:03:34.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:34.664880 14284 mini_oidc.cc:186] Starting OIDC Discovery server
02:03:34.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:34.665603 14284 webserver.cc:534] Webserver started at http://127.0.0.1:42573/ using document root <none> and password file <none>
02:03:34.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-07-08T02:03:34Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
02:03:34.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-07-08T02:03:34Z Disabled control of system clock
02:03:34.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:34.700270 14284 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu
02:03:34.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu
02:03:34.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/master-0/wal
02:03:34.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/master-0/data
02:03:34.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:03:34.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:03:34.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:03:34.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:03:34.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:03:34.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:03:34.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/master-0/logs
02:03:34.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/master-0/data/info.pb
02:03:34.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:03:34.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:03:34.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:03:34.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:03:34.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:03:34.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:03:34.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:03:34.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:03:34.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
02:03:34.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:03:34.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
02:03:34.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
02:03:34.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.13.243.62:45883
02:03:34.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.13.243.62
02:03:34.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:03:34.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.13.243.20:34675
02:03:34.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:03:34.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:03:34.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:03:34.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
02:03:34.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.13.243.62:45883,127.13.243.61:44961,127.13.243.60:35015
02:03:34.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
02:03:34.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:35485/jwks.json with env {}
02:03:35.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:35.023782 14295 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:03:35.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:35.024159 14295 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:03:35.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:35.024276 14295 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:03:35.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:35.032883 14295 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
02:03:35.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:35.033030 14295 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:03:35.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:35.033123 14295 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
02:03:35.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:35.033190 14295 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
02:03:35.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:35.033265 14295 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
02:03:35.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:35.033314 14295 flags.cc:432] Enabled experimental flag: --jwks_url=http://localhost:35485/jwks.json
02:03:35.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.044056 14295 master_runner.cc:387] Master server non-default flags:
02:03:35.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:03:35.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.13.243.20:34675
02:03:35.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:03:35.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:03:35.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/master-0/data
02:03:35.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/master-0/wal
02:03:35.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
02:03:35.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.13.243.62:45883,127.13.243.61:44961,127.13.243.60:35015
02:03:35.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:03:35.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:03:35.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
02:03:35.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.13.243.62:45883
02:03:35.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
02:03:35.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:03:35.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
02:03:35.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:35485/jwks.json
02:03:35.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:03:35.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:03:35.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/master-0/data/info.pb
02:03:35.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.13.243.62
02:03:35.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:03:35.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:03:35.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:03:35.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:03:35.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:03:35.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:03:35.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/master-0/logs
02:03:35.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:03:35.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:03:35.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:03:35.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
02:03:35.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:03:35.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision a3f35afc2b6dad6aa403ec58378221544a26edd0
02:03:35.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:03:35.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:40:41 UTC on bdcb31816ec0
02:03:35.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 16849
02:03:35.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
02:03:35.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.046430 14295 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:03:35.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.049131 14295 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:03:35.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:35.059419 14301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:35.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:35.059494 14300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:35.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:35.059638 14303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:35.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.060599 14295 server_base.cc:1061] running on GCE node
02:03:35.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.061497 14295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:03:35.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.063308 14295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:03:35.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.064543 14295 hybrid_clock.cc:648] HybridClock initialized: now 1783476215064476 us; error 56 us; skew 500 ppm
02:03:35.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.064975 14295 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:03:35.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.067376 14295 webserver.cc:534] Webserver started at http://127.13.243.62:40181/ using document root <none> and password file <none>
02:03:35.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.068197 14295 fs_manager.cc:362] Metadata directory not provided
02:03:35.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.068328 14295 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:03:35.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.068770 14295 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
02:03:35.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.071265 14295 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/master-0/data/instance:
02:03:35.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "b941c0b15a724504bfb3afd06fcd66fa"
02:03:35.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:03:35 on dist-test-slave-pjt5"
02:03:35.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.072175 14295 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/master-0/wal/instance:
02:03:35.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "b941c0b15a724504bfb3afd06fcd66fa"
02:03:35.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:03:35 on dist-test-slave-pjt5"
02:03:35.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.077883 14295 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.000s
02:03:35.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.081370 14309 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:03:35.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.082858 14295 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
02:03:35.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.082989 14295 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/master-0/data,/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/master-0/wal
02:03:35.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "b941c0b15a724504bfb3afd06fcd66fa"
02:03:35.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:03:35 on dist-test-slave-pjt5"
02:03:35.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.083221 14295 fs_report.cc:389] FS layout report
02:03:35.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:03:35.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/master-0/wal
02:03:35.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/master-0/wal
02:03:35.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/master-0/data/data
02:03:35.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:03:35.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:03:35.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:03:35.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:03:35.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:03:35.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:03:35.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:03:35.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:03:35.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:03:35.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:03:35.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:03:35.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.099360 14295 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:03:35.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.100198 14295 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:03:35.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.100418 14295 kserver.cc:163] Server-wide thread pool size limit: 3276
02:03:35.121 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.121130 14295 rpc_server.cc:307] RPC server started. Bound to: 127.13.243.62:45883
02:03:35.121 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.121153 14360 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.243.62:45883 every 8 connection(s)
02:03:35.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.122596 14295 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/master-0/data/info.pb
02:03:35.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.126885 14361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
02:03:35.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.128219 14284 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu as pid 14295
02:03:35.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.128477 14284 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/master-0/wal/instance
02:03:35.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.130622 14284 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu
02:03:35.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu
02:03:35.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/master-1/wal
02:03:35.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/master-1/data
02:03:35.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:03:35.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:03:35.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:03:35.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:03:35.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:03:35.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:03:35.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/master-1/logs
02:03:35.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/master-1/data/info.pb
02:03:35.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:03:35.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:03:35.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:03:35.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:03:35.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:03:35.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:03:35.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:03:35.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:03:35.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
02:03:35.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:03:35.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
02:03:35.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
02:03:35.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.13.243.61:44961
02:03:35.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.13.243.61
02:03:35.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:03:35.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.13.243.20:34675
02:03:35.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:03:35.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:03:35.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:03:35.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
02:03:35.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.13.243.62:45883,127.13.243.61:44961,127.13.243.60:35015
02:03:35.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
02:03:35.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:35485/jwks.json with env {}
02:03:35.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.134004 14361 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.243.62" port: 45883 } has no permanent_uuid. Determining permanent_uuid...
02:03:35.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.151386 14361 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.243.61" port: 44961 } has no permanent_uuid. Determining permanent_uuid...
02:03:35.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:35.152704 14312 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.243.61:44961: connect: Connection refused (error 111)
02:03:35.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:35.154676 14361 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.243.61:44961: Network error: Client connection negotiation failed: client connection to 127.13.243.61:44961: connect: Connection refused (error 111)
02:03:35.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.204097 14361 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.243.61" port: 44961 } attempt: 1
02:03:35.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:35.206501 14361 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.243.61:44961: Network error: Client connection negotiation failed: client connection to 127.13.243.61:44961: connect: Connection refused (error 111)
02:03:35.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.274765 14361 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.243.61" port: 44961 } attempt: 2
02:03:35.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:35.276993 14361 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.243.61:44961: Network error: Client connection negotiation failed: client connection to 127.13.243.61:44961: connect: Connection refused (error 111)
02:03:35.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.368286 14361 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.243.61" port: 44961 } attempt: 3
02:03:35.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:35.372260 14361 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.243.61:44961: Network error: Client connection negotiation failed: client connection to 127.13.243.61:44961: connect: Connection refused (error 111)
02:03:35.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:35.468886 14363 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:03:35.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:35.469187 14363 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:03:35.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:35.469266 14363 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:03:35.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:35.477160 14363 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
02:03:35.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:35.477243 14363 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:03:35.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:35.477300 14363 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
02:03:35.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:35.477341 14363 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
02:03:35.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:35.477394 14363 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
02:03:35.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:35.477429 14363 flags.cc:432] Enabled experimental flag: --jwks_url=http://localhost:35485/jwks.json
02:03:35.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.487073 14363 master_runner.cc:387] Master server non-default flags:
02:03:35.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:03:35.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.13.243.20:34675
02:03:35.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:03:35.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:03:35.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/master-1/data
02:03:35.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/master-1/wal
02:03:35.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
02:03:35.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.13.243.62:45883,127.13.243.61:44961,127.13.243.60:35015
02:03:35.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:03:35.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:03:35.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
02:03:35.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.13.243.61:44961
02:03:35.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
02:03:35.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:03:35.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
02:03:35.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:35485/jwks.json
02:03:35.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:03:35.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:03:35.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/master-1/data/info.pb
02:03:35.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.13.243.61
02:03:35.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:03:35.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:03:35.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:03:35.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:03:35.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:03:35.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:03:35.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/master-1/logs
02:03:35.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:03:35.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:03:35.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:03:35.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
02:03:35.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:03:35.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision a3f35afc2b6dad6aa403ec58378221544a26edd0
02:03:35.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:03:35.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:40:41 UTC on bdcb31816ec0
02:03:35.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 16849
02:03:35.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
02:03:35.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.488976 14363 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:03:35.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.491384 14363 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:03:35.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:35.500732 14371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:35.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:35.500885 14372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:35.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:35.503175 14374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:35.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.503614 14363 server_base.cc:1061] running on GCE node
02:03:35.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.505071 14363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:03:35.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.506484 14363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:03:35.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.507712 14363 hybrid_clock.cc:648] HybridClock initialized: now 1783476215507652 us; error 52 us; skew 500 ppm
02:03:35.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.508150 14363 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:03:35.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.510625 14363 webserver.cc:534] Webserver started at http://127.13.243.61:33995/ using document root <none> and password file <none>
02:03:35.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.511518 14363 fs_manager.cc:362] Metadata directory not provided
02:03:35.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.511647 14363 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:03:35.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.512111 14363 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
02:03:35.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.514590 14363 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/master-1/data/instance:
02:03:35.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "3abc49041df14c1a9406a684f33d7ed4"
02:03:35.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:03:35 on dist-test-slave-pjt5"
02:03:35.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.515513 14361 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.243.61" port: 44961 } attempt: 4
02:03:35.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.515957 14363 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/master-1/wal/instance:
02:03:35.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "3abc49041df14c1a9406a684f33d7ed4"
02:03:35.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:03:35 on dist-test-slave-pjt5"
02:03:35.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:35.518119 14361 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.243.61:44961: Network error: Client connection negotiation failed: client connection to 127.13.243.61:44961: connect: Connection refused (error 111)
02:03:35.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.521847 14363 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
02:03:35.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.525302 14380 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:03:35.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.526717 14363 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
02:03:35.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.526880 14363 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/master-1/data,/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/master-1/wal
02:03:35.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "3abc49041df14c1a9406a684f33d7ed4"
02:03:35.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:03:35 on dist-test-slave-pjt5"
02:03:35.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.527038 14363 fs_report.cc:389] FS layout report
02:03:35.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:03:35.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/master-1/wal
02:03:35.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/master-1/wal
02:03:35.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/master-1/data/data
02:03:35.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:03:35.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:03:35.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:03:35.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:03:35.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:03:35.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:03:35.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:03:35.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:03:35.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:03:35.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:03:35.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:03:35.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.553361 14363 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:03:35.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.554306 14363 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:03:35.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.554667 14363 kserver.cc:163] Server-wide thread pool size limit: 3276
02:03:35.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.573904 14363 rpc_server.cc:307] RPC server started. Bound to: 127.13.243.61:44961
02:03:35.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.573998 14431 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.243.61:44961 every 8 connection(s)
02:03:35.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.575364 14363 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/master-1/data/info.pb
02:03:35.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.579782 14432 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
02:03:35.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.580557 14284 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu as pid 14363
02:03:35.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.580749 14284 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/master-1/wal/instance
02:03:35.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.582193 14284 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu
02:03:35.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu
02:03:35.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/master-2/wal
02:03:35.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/master-2/data
02:03:35.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:03:35.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:03:35.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:03:35.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:03:35.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:03:35.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:03:35.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/master-2/logs
02:03:35.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/master-2/data/info.pb
02:03:35.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:03:35.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:03:35.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:03:35.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:03:35.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:03:35.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:03:35.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:03:35.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:03:35.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
02:03:35.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:03:35.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
02:03:35.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
02:03:35.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.13.243.60:35015
02:03:35.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.13.243.60
02:03:35.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:03:35.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.13.243.20:34675
02:03:35.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:03:35.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:03:35.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:03:35.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
02:03:35.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.13.243.62:45883,127.13.243.61:44961,127.13.243.60:35015
02:03:35.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
02:03:35.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:35485/jwks.json with env {}
02:03:35.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.589614 14432 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.243.62" port: 45883 } has no permanent_uuid. Determining permanent_uuid...
02:03:35.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.611025 14432 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.243.61" port: 44961 } has no permanent_uuid. Determining permanent_uuid...
02:03:35.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.620149 14432 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.243.60" port: 35015 } has no permanent_uuid. Determining permanent_uuid...
02:03:35.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:35.621428 14382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.243.60:35015: connect: Connection refused (error 111)
02:03:35.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:35.623103 14432 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.243.60:35015: Network error: Client connection negotiation failed: client connection to 127.13.243.60:35015: connect: Connection refused (error 111)
02:03:35.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.672405 14432 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.243.60" port: 35015 } attempt: 1
02:03:35.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:35.674530 14432 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.243.60:35015: Network error: Client connection negotiation failed: client connection to 127.13.243.60:35015: connect: Connection refused (error 111)
02:03:35.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.742758 14432 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.243.60" port: 35015 } attempt: 2
02:03:35.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:35.745042 14432 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.243.60:35015: Network error: Client connection negotiation failed: client connection to 127.13.243.60:35015: connect: Connection refused (error 111)
02:03:35.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.817426 14361 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.243.61" port: 44961 } attempt: 5
02:03:35.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.824896 14361 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.243.60" port: 35015 } has no permanent_uuid. Determining permanent_uuid...
02:03:35.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:35.827176 14361 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.243.60:35015: Network error: Client connection negotiation failed: client connection to 127.13.243.60:35015: connect: Connection refused (error 111)
02:03:35.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.836302 14432 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.243.60" port: 35015 } attempt: 3
02:03:35.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:35.838313 14432 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.243.60:35015: Network error: Client connection negotiation failed: client connection to 127.13.243.60:35015: connect: Connection refused (error 111)
02:03:35.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.878436 14361 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.243.60" port: 35015 } attempt: 1
02:03:35.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:35.880597 14361 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.243.60:35015: Network error: Client connection negotiation failed: client connection to 127.13.243.60:35015: connect: Connection refused (error 111)
02:03:35.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:35.916877 14434 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:03:35.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:35.917194 14434 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:03:35.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:35.917279 14434 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:03:35.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:35.925293 14434 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
02:03:35.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:35.925383 14434 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:03:35.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:35.925438 14434 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
02:03:35.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:35.925480 14434 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
02:03:35.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:35.925532 14434 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
02:03:35.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:35.925565 14434 flags.cc:432] Enabled experimental flag: --jwks_url=http://localhost:35485/jwks.json
02:03:35.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.935606 14434 master_runner.cc:387] Master server non-default flags:
02:03:35.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:03:35.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.13.243.20:34675
02:03:35.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:03:35.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:03:35.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/master-2/data
02:03:35.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/master-2/wal
02:03:35.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
02:03:35.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.13.243.62:45883,127.13.243.61:44961,127.13.243.60:35015
02:03:35.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:03:35.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:03:35.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
02:03:35.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.13.243.60:35015
02:03:35.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
02:03:35.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:03:35.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
02:03:35.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:35485/jwks.json
02:03:35.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:03:35.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:03:35.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/master-2/data/info.pb
02:03:35.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.13.243.60
02:03:35.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:03:35.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:03:35.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:03:35.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:03:35.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:03:35.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:03:35.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/master-2/logs
02:03:35.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:03:35.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:03:35.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:03:35.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
02:03:35.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:03:35.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision a3f35afc2b6dad6aa403ec58378221544a26edd0
02:03:35.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:03:35.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:40:41 UTC on bdcb31816ec0
02:03:35.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 16849
02:03:35.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
02:03:35.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.937517 14434 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:03:35.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.939965 14434 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:03:35.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:35.948482 14443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:35.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:35.948525 14442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:35.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.948808 14361 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.243.60" port: 35015 } attempt: 2
02:03:35.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:35.949194 14445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:35.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.950404 14434 server_base.cc:1061] running on GCE node
02:03:35.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:35.951023 14361 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.243.60:35015: Network error: Client connection negotiation failed: client connection to 127.13.243.60:35015: connect: Connection refused (error 111)
02:03:35.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.951306 14434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:03:35.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.952656 14434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:03:35.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.953856 14434 hybrid_clock.cc:648] HybridClock initialized: now 1783476215953799 us; error 47 us; skew 500 ppm
02:03:35.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.954186 14434 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:03:35.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.956634 14434 webserver.cc:534] Webserver started at http://127.13.243.60:39635/ using document root <none> and password file <none>
02:03:35.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.957309 14434 fs_manager.cc:362] Metadata directory not provided
02:03:35.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.957393 14434 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:03:35.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.957679 14434 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
02:03:35.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.959983 14434 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/master-2/data/instance:
02:03:35.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "3527c530b7bc44f1895e6d4232e34cba"
02:03:35.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:03:35 on dist-test-slave-pjt5"
02:03:35.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.960714 14434 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/master-2/wal/instance:
02:03:35.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "3527c530b7bc44f1895e6d4232e34cba"
02:03:35.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:03:35 on dist-test-slave-pjt5"
02:03:35.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.966004 14434 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.002s	sys 0.005s
02:03:35.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.969316 14451 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:03:35.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.970697 14434 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.003s
02:03:35.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.970834 14434 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/master-2/data,/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/master-2/wal
02:03:35.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "3527c530b7bc44f1895e6d4232e34cba"
02:03:35.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:03:35 on dist-test-slave-pjt5"
02:03:35.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.971031 14434 fs_report.cc:389] FS layout report
02:03:35.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:03:35.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/master-2/wal
02:03:35.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/master-2/wal
02:03:35.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/master-2/data/data
02:03:35.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:03:35.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:03:35.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:03:35.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:03:35.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:03:35.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:03:35.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:03:35.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:03:35.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:03:35.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:03:35.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:03:35.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.981565 14432 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.243.60" port: 35015 } attempt: 4
02:03:35.984 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:35.983721 14432 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.243.60:35015: Network error: Client connection negotiation failed: client connection to 127.13.243.60:35015: connect: Connection refused (error 111)
02:03:35.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.996022 14434 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:03:35.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.996832 14434 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:03:35.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:35.997079 14434 kserver.cc:163] Server-wide thread pool size limit: 3276
02:03:36.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.017210 14434 rpc_server.cc:307] RPC server started. Bound to: 127.13.243.60:35015
02:03:36.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.017232 14502 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.243.60:35015 every 8 connection(s)
02:03:36.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.018682 14434 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/master-2/data/info.pb
02:03:36.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.022512 14284 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu as pid 14434
02:03:36.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.022723 14284 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/master-2/wal/instance
02:03:36.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.023015 14503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
02:03:36.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.029801 14503 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.243.62" port: 45883 } has no permanent_uuid. Determining permanent_uuid...
02:03:36.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.043071 14503 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.243.61" port: 44961 } has no permanent_uuid. Determining permanent_uuid...
02:03:36.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.050683 14503 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.243.60" port: 35015 } has no permanent_uuid. Determining permanent_uuid...
02:03:36.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.057230 14361 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.243.60" port: 35015 } attempt: 3
02:03:36.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.064261 14503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3527c530b7bc44f1895e6d4232e34cba: Bootstrap starting.
02:03:36.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.067924 14503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3527c530b7bc44f1895e6d4232e34cba: Neither blocks nor log segments found. Creating new log.
02:03:36.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.068264 14361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b941c0b15a724504bfb3afd06fcd66fa: Bootstrap starting.
02:03:36.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.068974 14503 log.cc:826] T 00000000000000000000000000000000 P 3527c530b7bc44f1895e6d4232e34cba: Log is configured to *not* fsync() on all Append() calls
02:03:36.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.071537 14503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3527c530b7bc44f1895e6d4232e34cba: No bootstrap required, opened a new log
02:03:36.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.071619 14361 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b941c0b15a724504bfb3afd06fcd66fa: Neither blocks nor log segments found. Creating new log.
02:03:36.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.073613 14361 log.cc:826] T 00000000000000000000000000000000 P b941c0b15a724504bfb3afd06fcd66fa: Log is configured to *not* fsync() on all Append() calls
02:03:36.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.076002 14361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b941c0b15a724504bfb3afd06fcd66fa: No bootstrap required, opened a new log
02:03:36.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.076937 14503 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3527c530b7bc44f1895e6d4232e34cba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b941c0b15a724504bfb3afd06fcd66fa" member_type: VOTER last_known_addr { host: "127.13.243.62" port: 45883 } } peers { permanent_uuid: "3abc49041df14c1a9406a684f33d7ed4" member_type: VOTER last_known_addr { host: "127.13.243.61" port: 44961 } } peers { permanent_uuid: "3527c530b7bc44f1895e6d4232e34cba" member_type: VOTER last_known_addr { host: "127.13.243.60" port: 35015 } }
02:03:36.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.077186 14503 raft_consensus.cc:385] T 00000000000000000000000000000000 P 3527c530b7bc44f1895e6d4232e34cba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
02:03:36.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.077263 14503 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3527c530b7bc44f1895e6d4232e34cba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3527c530b7bc44f1895e6d4232e34cba, State: Initialized, Role: FOLLOWER
02:03:36.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.077946 14503 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3527c530b7bc44f1895e6d4232e34cba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b941c0b15a724504bfb3afd06fcd66fa" member_type: VOTER last_known_addr { host: "127.13.243.62" port: 45883 } } peers { permanent_uuid: "3abc49041df14c1a9406a684f33d7ed4" member_type: VOTER last_known_addr { host: "127.13.243.61" port: 44961 } } peers { permanent_uuid: "3527c530b7bc44f1895e6d4232e34cba" member_type: VOTER last_known_addr { host: "127.13.243.60" port: 35015 } }
02:03:36.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.079643 14513 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3527c530b7bc44f1895e6d4232e34cba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b941c0b15a724504bfb3afd06fcd66fa" member_type: VOTER last_known_addr { host: "127.13.243.62" port: 45883 } } peers { permanent_uuid: "3abc49041df14c1a9406a684f33d7ed4" member_type: VOTER last_known_addr { host: "127.13.243.61" port: 44961 } } peers { permanent_uuid: "3527c530b7bc44f1895e6d4232e34cba" member_type: VOTER last_known_addr { host: "127.13.243.60" port: 35015 } } }
02:03:36.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.080336 14513 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3527c530b7bc44f1895e6d4232e34cba [sys.catalog]: This master's current role is: FOLLOWER
02:03:36.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.080834 14503 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3527c530b7bc44f1895e6d4232e34cba [sys.catalog]: configured and running, proceeding with master startup.
02:03:36.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.082394 14361 raft_consensus.cc:359] T 00000000000000000000000000000000 P b941c0b15a724504bfb3afd06fcd66fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b941c0b15a724504bfb3afd06fcd66fa" member_type: VOTER last_known_addr { host: "127.13.243.62" port: 45883 } } peers { permanent_uuid: "3abc49041df14c1a9406a684f33d7ed4" member_type: VOTER last_known_addr { host: "127.13.243.61" port: 44961 } } peers { permanent_uuid: "3527c530b7bc44f1895e6d4232e34cba" member_type: VOTER last_known_addr { host: "127.13.243.60" port: 35015 } }
02:03:36.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.082971 14361 raft_consensus.cc:385] T 00000000000000000000000000000000 P b941c0b15a724504bfb3afd06fcd66fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
02:03:36.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.083345 14361 raft_consensus.cc:740] T 00000000000000000000000000000000 P b941c0b15a724504bfb3afd06fcd66fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b941c0b15a724504bfb3afd06fcd66fa, State: Initialized, Role: FOLLOWER
02:03:36.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.084002 14361 consensus_queue.cc:260] T 00000000000000000000000000000000 P b941c0b15a724504bfb3afd06fcd66fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b941c0b15a724504bfb3afd06fcd66fa" member_type: VOTER last_known_addr { host: "127.13.243.62" port: 45883 } } peers { permanent_uuid: "3abc49041df14c1a9406a684f33d7ed4" member_type: VOTER last_known_addr { host: "127.13.243.61" port: 44961 } } peers { permanent_uuid: "3527c530b7bc44f1895e6d4232e34cba" member_type: VOTER last_known_addr { host: "127.13.243.60" port: 35015 } }
02:03:36.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.084159 14513 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3527c530b7bc44f1895e6d4232e34cba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
02:03:36.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.084443 14513 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3527c530b7bc44f1895e6d4232e34cba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b941c0b15a724504bfb3afd06fcd66fa" member_type: VOTER last_known_addr { host: "127.13.243.62" port: 45883 } } peers { permanent_uuid: "3abc49041df14c1a9406a684f33d7ed4" member_type: VOTER last_known_addr { host: "127.13.243.61" port: 44961 } } peers { permanent_uuid: "3527c530b7bc44f1895e6d4232e34cba" member_type: VOTER last_known_addr { host: "127.13.243.60" port: 35015 } }
02:03:36.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.085476 14522 sys_catalog.cc:455] T 00000000000000000000000000000000 P b941c0b15a724504bfb3afd06fcd66fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b941c0b15a724504bfb3afd06fcd66fa" member_type: VOTER last_known_addr { host: "127.13.243.62" port: 45883 } } peers { permanent_uuid: "3abc49041df14c1a9406a684f33d7ed4" member_type: VOTER last_known_addr { host: "127.13.243.61" port: 44961 } } peers { permanent_uuid: "3527c530b7bc44f1895e6d4232e34cba" member_type: VOTER last_known_addr { host: "127.13.243.60" port: 35015 } } }
02:03:36.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.087904 14513 leader_election.cc:290] T 00000000000000000000000000000000 P 3527c530b7bc44f1895e6d4232e34cba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b941c0b15a724504bfb3afd06fcd66fa (127.13.243.62:45883), 3abc49041df14c1a9406a684f33d7ed4 (127.13.243.61:44961)
02:03:36.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.086860 14361 sys_catalog.cc:565] T 00000000000000000000000000000000 P b941c0b15a724504bfb3afd06fcd66fa [sys.catalog]: configured and running, proceeding with master startup.
02:03:36.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.087427 14336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3527c530b7bc44f1895e6d4232e34cba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b941c0b15a724504bfb3afd06fcd66fa" is_pre_election: true
02:03:36.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.092154 14336 raft_consensus.cc:2468] T 00000000000000000000000000000000 P b941c0b15a724504bfb3afd06fcd66fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3527c530b7bc44f1895e6d4232e34cba in term 0.
02:03:36.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.092835 14455 leader_election.cc:304] T 00000000000000000000000000000000 P 3527c530b7bc44f1895e6d4232e34cba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3527c530b7bc44f1895e6d4232e34cba, b941c0b15a724504bfb3afd06fcd66fa; no voters: 
02:03:36.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:36.094146 14525 catalog_manager.cc:1593] T 00000000000000000000000000000000 P 3527c530b7bc44f1895e6d4232e34cba: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
02:03:36.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:36.094260 14525 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
02:03:36.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.091795 14434 master_runner.cc:187] Error getting master registration for 127.13.243.62:45883: OK, instance_id { permanent_uuid: "b941c0b15a724504bfb3afd06fcd66fa" instance_seqno: 1783476215115564 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
02:03:36.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.097147 14434 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
02:03:36.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.097303 14513 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3527c530b7bc44f1895e6d4232e34cba [term 0 FOLLOWER]: Leader pre-election won for term 1
02:03:36.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.097458 14513 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3527c530b7bc44f1895e6d4232e34cba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
02:03:36.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.097554 14513 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3527c530b7bc44f1895e6d4232e34cba [term 0 FOLLOWER]: Advancing to term 1
02:03:36.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.100605 14513 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3527c530b7bc44f1895e6d4232e34cba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b941c0b15a724504bfb3afd06fcd66fa" member_type: VOTER last_known_addr { host: "127.13.243.62" port: 45883 } } peers { permanent_uuid: "3abc49041df14c1a9406a684f33d7ed4" member_type: VOTER last_known_addr { host: "127.13.243.61" port: 44961 } } peers { permanent_uuid: "3527c530b7bc44f1895e6d4232e34cba" member_type: VOTER last_known_addr { host: "127.13.243.60" port: 35015 } }
02:03:36.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.101357 14513 leader_election.cc:290] T 00000000000000000000000000000000 P 3527c530b7bc44f1895e6d4232e34cba [CANDIDATE]: Term 1 election: Requested vote from peers b941c0b15a724504bfb3afd06fcd66fa (127.13.243.62:45883), 3abc49041df14c1a9406a684f33d7ed4 (127.13.243.61:44961)
02:03:36.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.102093 14336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3527c530b7bc44f1895e6d4232e34cba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b941c0b15a724504bfb3afd06fcd66fa"
02:03:36.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.102299 14336 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b941c0b15a724504bfb3afd06fcd66fa [term 0 FOLLOWER]: Advancing to term 1
02:03:36.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.105631 14336 raft_consensus.cc:2468] T 00000000000000000000000000000000 P b941c0b15a724504bfb3afd06fcd66fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3527c530b7bc44f1895e6d4232e34cba in term 1.
02:03:36.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:36.106042 14315 tablet.cc:2406] T 00000000000000000000000000000000 P b941c0b15a724504bfb3afd06fcd66fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
02:03:36.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.106177 14455 leader_election.cc:304] T 00000000000000000000000000000000 P 3527c530b7bc44f1895e6d4232e34cba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3527c530b7bc44f1895e6d4232e34cba, b941c0b15a724504bfb3afd06fcd66fa; no voters: 
02:03:36.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.106501 14513 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3527c530b7bc44f1895e6d4232e34cba [term 1 FOLLOWER]: Leader election won for term 1
02:03:36.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.107933 14513 raft_consensus.cc:697] T 00000000000000000000000000000000 P 3527c530b7bc44f1895e6d4232e34cba [term 1 LEADER]: Becoming Leader. State: Replica: 3527c530b7bc44f1895e6d4232e34cba, State: Running, Role: LEADER
02:03:36.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.108480 14513 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3527c530b7bc44f1895e6d4232e34cba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b941c0b15a724504bfb3afd06fcd66fa" member_type: VOTER last_known_addr { host: "127.13.243.62" port: 45883 } } peers { permanent_uuid: "3abc49041df14c1a9406a684f33d7ed4" member_type: VOTER last_known_addr { host: "127.13.243.61" port: 44961 } } peers { permanent_uuid: "3527c530b7bc44f1895e6d4232e34cba" member_type: VOTER last_known_addr { host: "127.13.243.60" port: 35015 } }
02:03:36.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.110596 14522 sys_catalog.cc:458] T 00000000000000000000000000000000 P b941c0b15a724504bfb3afd06fcd66fa [sys.catalog]: This master's current role is: FOLLOWER
02:03:36.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:36.133988 14539 catalog_manager.cc:1593] T 00000000000000000000000000000000 P b941c0b15a724504bfb3afd06fcd66fa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
02:03:36.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:36.134750 14539 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
02:03:36.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.135003 14528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3527c530b7bc44f1895e6d4232e34cba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3527c530b7bc44f1895e6d4232e34cba. Latest consensus state: current_term: 1 leader_uuid: "3527c530b7bc44f1895e6d4232e34cba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b941c0b15a724504bfb3afd06fcd66fa" member_type: VOTER last_known_addr { host: "127.13.243.62" port: 45883 } } peers { permanent_uuid: "3abc49041df14c1a9406a684f33d7ed4" member_type: VOTER last_known_addr { host: "127.13.243.61" port: 44961 } } peers { permanent_uuid: "3527c530b7bc44f1895e6d4232e34cba" member_type: VOTER last_known_addr { host: "127.13.243.60" port: 35015 } } }
02:03:36.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.135388 14528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3527c530b7bc44f1895e6d4232e34cba [sys.catalog]: This master's current role is: LEADER
02:03:36.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.133186 14295 master_runner.cc:187] Error getting master registration for 127.13.243.61:44961: OK, instance_id { permanent_uuid: "3abc49041df14c1a9406a684f33d7ed4" instance_seqno: 1783476215568795 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
02:03:36.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.140920 14295 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
02:03:36.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.143354 14541 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
02:03:36.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.147298 14541 catalog_manager.cc:1519] Initializing Kudu cluster ID...
02:03:36.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.157958 14336 raft_consensus.cc:1275] T 00000000000000000000000000000000 P b941c0b15a724504bfb3afd06fcd66fa [term 1 FOLLOWER]: Refusing update from remote peer 3527c530b7bc44f1895e6d4232e34cba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
02:03:36.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.158946 14528 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 3527c530b7bc44f1895e6d4232e34cba [LEADER]: Connected to new peer: Peer: permanent_uuid: "b941c0b15a724504bfb3afd06fcd66fa" member_type: VOTER last_known_addr { host: "127.13.243.62" port: 45883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
02:03:36.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.178144 14522 sys_catalog.cc:455] T 00000000000000000000000000000000 P b941c0b15a724504bfb3afd06fcd66fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3527c530b7bc44f1895e6d4232e34cba. Latest consensus state: current_term: 1 leader_uuid: "3527c530b7bc44f1895e6d4232e34cba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b941c0b15a724504bfb3afd06fcd66fa" member_type: VOTER last_known_addr { host: "127.13.243.62" port: 45883 } } peers { permanent_uuid: "3abc49041df14c1a9406a684f33d7ed4" member_type: VOTER last_known_addr { host: "127.13.243.61" port: 44961 } } peers { permanent_uuid: "3527c530b7bc44f1895e6d4232e34cba" member_type: VOTER last_known_addr { host: "127.13.243.60" port: 35015 } } }
02:03:36.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.178922 14522 sys_catalog.cc:458] T 00000000000000000000000000000000 P b941c0b15a724504bfb3afd06fcd66fa [sys.catalog]: This master's current role is: FOLLOWER
02:03:36.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.183682 14522 sys_catalog.cc:455] T 00000000000000000000000000000000 P b941c0b15a724504bfb3afd06fcd66fa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3527c530b7bc44f1895e6d4232e34cba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b941c0b15a724504bfb3afd06fcd66fa" member_type: VOTER last_known_addr { host: "127.13.243.62" port: 45883 } } peers { permanent_uuid: "3abc49041df14c1a9406a684f33d7ed4" member_type: VOTER last_known_addr { host: "127.13.243.61" port: 44961 } } peers { permanent_uuid: "3527c530b7bc44f1895e6d4232e34cba" member_type: VOTER last_known_addr { host: "127.13.243.60" port: 35015 } } }
02:03:36.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.183887 14522 sys_catalog.cc:458] T 00000000000000000000000000000000 P b941c0b15a724504bfb3afd06fcd66fa [sys.catalog]: This master's current role is: FOLLOWER
02:03:36.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.196918 14528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3527c530b7bc44f1895e6d4232e34cba [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3527c530b7bc44f1895e6d4232e34cba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b941c0b15a724504bfb3afd06fcd66fa" member_type: VOTER last_known_addr { host: "127.13.243.62" port: 45883 } } peers { permanent_uuid: "3abc49041df14c1a9406a684f33d7ed4" member_type: VOTER last_known_addr { host: "127.13.243.61" port: 44961 } } peers { permanent_uuid: "3527c530b7bc44f1895e6d4232e34cba" member_type: VOTER last_known_addr { host: "127.13.243.60" port: 35015 } } }
02:03:36.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.197235 14528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3527c530b7bc44f1895e6d4232e34cba [sys.catalog]: This master's current role is: LEADER
02:03:36.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.199015 14541 catalog_manager.cc:1382] Generated new cluster ID: b6b271981afd4cf29d06d58794657092
02:03:36.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.199126 14541 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
02:03:36.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.222497 14541 catalog_manager.cc:1405] Generated new certificate authority record
02:03:36.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.224256 14541 catalog_manager.cc:1539] Loading token signing keys...
02:03:36.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.246474 14541 catalog_manager.cc:6069] T 00000000000000000000000000000000 P 3527c530b7bc44f1895e6d4232e34cba: Generated new TSK 0
02:03:36.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.248148 14541 catalog_manager.cc:1549] Initializing in-progress tserver states...
02:03:36.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.273368 14284 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu
02:03:36.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu
02:03:36.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-0/wal
02:03:36.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-0/data
02:03:36.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:03:36.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:03:36.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:03:36.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:03:36.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:03:36.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:03:36.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-0/logs
02:03:36.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-0/data/info.pb
02:03:36.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:03:36.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:03:36.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:03:36.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:03:36.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:03:36.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:03:36.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:03:36.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:03:36.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:03:36.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:03:36.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.13.243.1:0
02:03:36.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.13.243.1
02:03:36.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.13.243.1
02:03:36.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:03:36.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.13.243.62:45883,127.13.243.61:44961,127.13.243.60:35015
02:03:36.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.13.243.20:34675
02:03:36.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:03:36.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:03:36.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:03:36.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.282922 14432 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.243.60" port: 35015 } attempt: 5
02:03:36.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.298128 14432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3abc49041df14c1a9406a684f33d7ed4: Bootstrap starting.
02:03:36.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.307065 14432 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3abc49041df14c1a9406a684f33d7ed4: Neither blocks nor log segments found. Creating new log.
02:03:36.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.310541 14432 log.cc:826] T 00000000000000000000000000000000 P 3abc49041df14c1a9406a684f33d7ed4: Log is configured to *not* fsync() on all Append() calls
02:03:36.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.326153 14432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3abc49041df14c1a9406a684f33d7ed4: No bootstrap required, opened a new log
02:03:36.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.342631 14432 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3abc49041df14c1a9406a684f33d7ed4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b941c0b15a724504bfb3afd06fcd66fa" member_type: VOTER last_known_addr { host: "127.13.243.62" port: 45883 } } peers { permanent_uuid: "3abc49041df14c1a9406a684f33d7ed4" member_type: VOTER last_known_addr { host: "127.13.243.61" port: 44961 } } peers { permanent_uuid: "3527c530b7bc44f1895e6d4232e34cba" member_type: VOTER last_known_addr { host: "127.13.243.60" port: 35015 } }
02:03:36.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.343796 14432 raft_consensus.cc:385] T 00000000000000000000000000000000 P 3abc49041df14c1a9406a684f33d7ed4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
02:03:36.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.344388 14432 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3abc49041df14c1a9406a684f33d7ed4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3abc49041df14c1a9406a684f33d7ed4, State: Initialized, Role: FOLLOWER
02:03:36.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.346174 14432 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3abc49041df14c1a9406a684f33d7ed4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b941c0b15a724504bfb3afd06fcd66fa" member_type: VOTER last_known_addr { host: "127.13.243.62" port: 45883 } } peers { permanent_uuid: "3abc49041df14c1a9406a684f33d7ed4" member_type: VOTER last_known_addr { host: "127.13.243.61" port: 44961 } } peers { permanent_uuid: "3527c530b7bc44f1895e6d4232e34cba" member_type: VOTER last_known_addr { host: "127.13.243.60" port: 35015 } }
02:03:36.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.360361 14407 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3527c530b7bc44f1895e6d4232e34cba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3abc49041df14c1a9406a684f33d7ed4" is_pre_election: true
02:03:36.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.360774 14406 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3527c530b7bc44f1895e6d4232e34cba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3abc49041df14c1a9406a684f33d7ed4"
02:03:36.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.362818 14406 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3abc49041df14c1a9406a684f33d7ed4 [term 0 FOLLOWER]: Advancing to term 1
02:03:36.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.366180 14406 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 3abc49041df14c1a9406a684f33d7ed4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3527c530b7bc44f1895e6d4232e34cba in term 1.
02:03:36.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.367156 14405 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 3abc49041df14c1a9406a684f33d7ed4 [term 1 FOLLOWER]: Refusing update from remote peer 3527c530b7bc44f1895e6d4232e34cba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
02:03:36.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.368295 14548 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 3527c530b7bc44f1895e6d4232e34cba [LEADER]: Connected to new peer: Peer: permanent_uuid: "3abc49041df14c1a9406a684f33d7ed4" member_type: VOTER last_known_addr { host: "127.13.243.61" port: 44961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
02:03:36.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.369148 14407 raft_consensus.cc:2376] T 00000000000000000000000000000000 P 3abc49041df14c1a9406a684f33d7ed4 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 3527c530b7bc44f1895e6d4232e34cba in term 1. Re-sending same reply.
02:03:36.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.370770 14432 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3abc49041df14c1a9406a684f33d7ed4 [sys.catalog]: configured and running, proceeding with master startup.
02:03:36.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.421649 14552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3abc49041df14c1a9406a684f33d7ed4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3527c530b7bc44f1895e6d4232e34cba. Latest consensus state: current_term: 1 leader_uuid: "3527c530b7bc44f1895e6d4232e34cba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b941c0b15a724504bfb3afd06fcd66fa" member_type: VOTER last_known_addr { host: "127.13.243.62" port: 45883 } } peers { permanent_uuid: "3abc49041df14c1a9406a684f33d7ed4" member_type: VOTER last_known_addr { host: "127.13.243.61" port: 44961 } } peers { permanent_uuid: "3527c530b7bc44f1895e6d4232e34cba" member_type: VOTER last_known_addr { host: "127.13.243.60" port: 35015 } } }
02:03:36.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.421586 14551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3abc49041df14c1a9406a684f33d7ed4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3527c530b7bc44f1895e6d4232e34cba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b941c0b15a724504bfb3afd06fcd66fa" member_type: VOTER last_known_addr { host: "127.13.243.62" port: 45883 } } peers { permanent_uuid: "3abc49041df14c1a9406a684f33d7ed4" member_type: VOTER last_known_addr { host: "127.13.243.61" port: 44961 } } peers { permanent_uuid: "3527c530b7bc44f1895e6d4232e34cba" member_type: VOTER last_known_addr { host: "127.13.243.60" port: 35015 } } }
02:03:36.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.422034 14552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3abc49041df14c1a9406a684f33d7ed4 [sys.catalog]: This master's current role is: FOLLOWER
02:03:36.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.422042 14551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3abc49041df14c1a9406a684f33d7ed4 [sys.catalog]: This master's current role is: FOLLOWER
02:03:36.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.423339 14561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3abc49041df14c1a9406a684f33d7ed4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3527c530b7bc44f1895e6d4232e34cba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b941c0b15a724504bfb3afd06fcd66fa" member_type: VOTER last_known_addr { host: "127.13.243.62" port: 45883 } } peers { permanent_uuid: "3abc49041df14c1a9406a684f33d7ed4" member_type: VOTER last_known_addr { host: "127.13.243.61" port: 44961 } } peers { permanent_uuid: "3527c530b7bc44f1895e6d4232e34cba" member_type: VOTER last_known_addr { host: "127.13.243.60" port: 35015 } } }
02:03:36.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.423920 14561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3abc49041df14c1a9406a684f33d7ed4 [sys.catalog]: This master's current role is: FOLLOWER
02:03:36.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.439666 14569 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
02:03:36.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:36.770000 14549 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:03:36.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:36.770306 14549 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:03:36.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:36.770466 14549 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:03:36.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:36.778697 14549 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:03:36.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:36.778854 14549 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.243.1
02:03:36.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.787709 14548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3527c530b7bc44f1895e6d4232e34cba [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3527c530b7bc44f1895e6d4232e34cba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b941c0b15a724504bfb3afd06fcd66fa" member_type: VOTER last_known_addr { host: "127.13.243.62" port: 45883 } } peers { permanent_uuid: "3abc49041df14c1a9406a684f33d7ed4" member_type: VOTER last_known_addr { host: "127.13.243.61" port: 44961 } } peers { permanent_uuid: "3527c530b7bc44f1895e6d4232e34cba" member_type: VOTER last_known_addr { host: "127.13.243.60" port: 35015 } } }
02:03:36.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.787969 14548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3527c530b7bc44f1895e6d4232e34cba [sys.catalog]: This master's current role is: LEADER
02:03:36.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.789852 14549 tablet_server_runner.cc:78] Tablet server non-default flags:
02:03:36.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:03:36.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.13.243.20:34675
02:03:36.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:03:36.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:03:36.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-0/data
02:03:36.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-0/wal
02:03:36.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:03:36.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:03:36.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.13.243.1:0
02:03:36.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:03:36.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:03:36.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:03:36.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-0/data/info.pb
02:03:36.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.13.243.1
02:03:36.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:03:36.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.13.243.62:45883,127.13.243.61:44961,127.13.243.60:35015
02:03:36.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:03:36.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:03:36.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:03:36.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:03:36.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:03:36.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.13.243.1
02:03:36.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-0/logs
02:03:36.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:03:36.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:03:36.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:03:36.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:03:36.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:03:36.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision a3f35afc2b6dad6aa403ec58378221544a26edd0
02:03:36.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:03:36.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:40:41 UTC on bdcb31816ec0
02:03:36.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 16849
02:03:36.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
02:03:36.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.792488 14549 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:03:36.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.794744 14549 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:03:36.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:36.804782 14578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:36.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:36.805809 14577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:36.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:36.807921 14580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:36.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.808590 14549 server_base.cc:1061] running on GCE node
02:03:36.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.809248 14549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:03:36.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.810370 14549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:03:36.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.811599 14549 hybrid_clock.cc:648] HybridClock initialized: now 1783476216811541 us; error 49 us; skew 500 ppm
02:03:36.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.811985 14549 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:03:36.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.814383 14549 webserver.cc:534] Webserver started at http://127.13.243.1:42315/ using document root <none> and password file <none>
02:03:36.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.815196 14549 fs_manager.cc:362] Metadata directory not provided
02:03:36.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.815327 14549 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:03:36.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.815752 14549 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
02:03:36.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.818140 14549 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-0/data/instance:
02:03:36.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "6422ba2afbf742febb59b258c6ac5101"
02:03:36.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:03:36 on dist-test-slave-pjt5"
02:03:36.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.819072 14549 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-0/wal/instance:
02:03:36.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "6422ba2afbf742febb59b258c6ac5101"
02:03:36.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:03:36 on dist-test-slave-pjt5"
02:03:36.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.824496 14549 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.001s	sys 0.004s
02:03:36.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.828030 14586 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:03:36.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.829291 14549 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
02:03:36.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.829489 14549 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-0/data,/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-0/wal
02:03:36.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "6422ba2afbf742febb59b258c6ac5101"
02:03:36.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:03:36 on dist-test-slave-pjt5"
02:03:36.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.829756 14549 fs_report.cc:389] FS layout report
02:03:36.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:03:36.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-0/wal
02:03:36.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-0/wal
02:03:36.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-0/data/data
02:03:36.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:03:36.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:03:36.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:03:36.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:03:36.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:03:36.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:03:36.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:03:36.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:03:36.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:03:36.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:03:36.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:03:36.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.852736 14549 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:03:36.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.853746 14549 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:03:36.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.854066 14549 kserver.cc:163] Server-wide thread pool size limit: 3276
02:03:36.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.855214 14549 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:03:36.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.857019 14549 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
02:03:36.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.857141 14549 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
02:03:36.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.857242 14549 ts_tablet_manager.cc:616] Registered 0 tablets
02:03:36.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.857321 14549 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
02:03:36.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.890931 14549 rpc_server.cc:307] RPC server started. Bound to: 127.13.243.1:33421
02:03:36.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.890988 14698 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.243.1:33421 every 8 connection(s)
02:03:36.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.892632 14549 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-0/data/info.pb
02:03:36.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.894555 14284 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu as pid 14549
02:03:36.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.894755 14284 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-0/wal/instance
02:03:36.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.902189 14284 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu
02:03:36.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu
02:03:36.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-1/wal
02:03:36.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-1/data
02:03:36.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:03:36.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:03:36.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:03:36.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:03:36.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:03:36.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:03:36.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-1/logs
02:03:36.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-1/data/info.pb
02:03:36.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:03:36.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:03:36.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:03:36.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:03:36.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:03:36.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:03:36.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:03:36.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:03:36.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:03:36.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:03:36.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.13.243.2:0
02:03:36.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.13.243.2
02:03:36.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.13.243.2
02:03:36.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:03:36.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.13.243.62:45883,127.13.243.61:44961,127.13.243.60:35015
02:03:36.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.13.243.20:34675
02:03:36.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:03:36.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:03:36.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:03:36.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.923763 14701 heartbeater.cc:344] Connected to a master server at 127.13.243.61:44961
02:03:36.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.924206 14701 heartbeater.cc:461] Registering TS with master...
02:03:36.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.925133 14701 heartbeater.cc:507] Master 127.13.243.61:44961 requested a full tablet report, sending...
02:03:36.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.926676 14699 heartbeater.cc:344] Connected to a master server at 127.13.243.60:35015
02:03:36.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.926820 14699 heartbeater.cc:461] Registering TS with master...
02:03:36.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.927268 14699 heartbeater.cc:507] Master 127.13.243.60:35015 requested a full tablet report, sending...
02:03:36.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.927629 14397 ts_manager.cc:194] Registered new tserver with Master: 6422ba2afbf742febb59b258c6ac5101 (127.13.243.1:33421)
02:03:36.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.929502 14468 ts_manager.cc:194] Registered new tserver with Master: 6422ba2afbf742febb59b258c6ac5101 (127.13.243.1:33421)
02:03:36.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.931269 14700 heartbeater.cc:344] Connected to a master server at 127.13.243.62:45883
02:03:36.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.931432 14700 heartbeater.cc:461] Registering TS with master...
02:03:36.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.931897 14700 heartbeater.cc:507] Master 127.13.243.62:45883 requested a full tablet report, sending...
02:03:36.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.931929 14468 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.13.243.1:43865
02:03:36.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:36.933624 14326 ts_manager.cc:194] Registered new tserver with Master: 6422ba2afbf742febb59b258c6ac5101 (127.13.243.1:33421)
02:03:37.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.137059 14539 catalog_manager.cc:1294] Loaded cluster ID: b6b271981afd4cf29d06d58794657092
02:03:37.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.137197 14539 catalog_manager.cc:1587] T 00000000000000000000000000000000 P b941c0b15a724504bfb3afd06fcd66fa: loading cluster ID for follower catalog manager: success
02:03:37.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.140064 14539 catalog_manager.cc:1609] T 00000000000000000000000000000000 P b941c0b15a724504bfb3afd06fcd66fa: acquiring CA information for follower catalog manager: success
02:03:37.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.141963 14539 catalog_manager.cc:1637] T 00000000000000000000000000000000 P b941c0b15a724504bfb3afd06fcd66fa: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
02:03:37.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:37.248894 14707 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:03:37.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:37.249185 14707 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:03:37.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:37.249267 14707 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:03:37.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:37.257400 14707 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:03:37.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:37.257625 14707 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.243.2
02:03:37.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.267514 14707 tablet_server_runner.cc:78] Tablet server non-default flags:
02:03:37.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:03:37.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.13.243.20:34675
02:03:37.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:03:37.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:03:37.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-1/data
02:03:37.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-1/wal
02:03:37.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:03:37.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:03:37.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.13.243.2:0
02:03:37.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:03:37.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:03:37.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:03:37.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-1/data/info.pb
02:03:37.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.13.243.2
02:03:37.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:03:37.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.13.243.62:45883,127.13.243.61:44961,127.13.243.60:35015
02:03:37.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:03:37.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:03:37.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:03:37.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:03:37.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:03:37.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.13.243.2
02:03:37.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-1/logs
02:03:37.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:03:37.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:03:37.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:03:37.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:03:37.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:03:37.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision a3f35afc2b6dad6aa403ec58378221544a26edd0
02:03:37.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:03:37.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:40:41 UTC on bdcb31816ec0
02:03:37.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 16849
02:03:37.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
02:03:37.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.269073 14707 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:03:37.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.270874 14707 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:03:37.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:37.279649 14714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:37.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:37.280326 14715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:37.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:37.281272 14717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:37.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.281843 14707 server_base.cc:1061] running on GCE node
02:03:37.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.282502 14707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:03:37.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.283331 14707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:03:37.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.284530 14707 hybrid_clock.cc:648] HybridClock initialized: now 1783476217284481 us; error 40 us; skew 500 ppm
02:03:37.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.284795 14707 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:03:37.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.286870 14707 webserver.cc:534] Webserver started at http://127.13.243.2:42383/ using document root <none> and password file <none>
02:03:37.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.287618 14707 fs_manager.cc:362] Metadata directory not provided
02:03:37.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.287704 14707 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:03:37.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.288085 14707 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
02:03:37.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.290652 14707 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-1/data/instance:
02:03:37.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "b123247401d540528f8324235e936260"
02:03:37.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:03:37 on dist-test-slave-pjt5"
02:03:37.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.291518 14707 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-1/wal/instance:
02:03:37.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "b123247401d540528f8324235e936260"
02:03:37.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:03:37 on dist-test-slave-pjt5"
02:03:37.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.296972 14707 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
02:03:37.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.300536 14723 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:03:37.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.302016 14707 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
02:03:37.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.302158 14707 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-1/data,/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-1/wal
02:03:37.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "b123247401d540528f8324235e936260"
02:03:37.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:03:37 on dist-test-slave-pjt5"
02:03:37.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.302439 14707 fs_report.cc:389] FS layout report
02:03:37.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:03:37.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-1/wal
02:03:37.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-1/wal
02:03:37.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-1/data/data
02:03:37.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:03:37.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:03:37.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:03:37.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:03:37.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:03:37.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:03:37.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:03:37.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:03:37.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:03:37.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:03:37.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:03:37.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.324030 14707 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:03:37.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.324882 14707 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:03:37.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.325127 14707 kserver.cc:163] Server-wide thread pool size limit: 3276
02:03:37.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.326045 14707 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:03:37.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.328052 14707 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
02:03:37.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.328157 14707 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
02:03:37.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.328233 14707 ts_tablet_manager.cc:616] Registered 0 tablets
02:03:37.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.328284 14707 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
02:03:37.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.363516 14707 rpc_server.cc:307] RPC server started. Bound to: 127.13.243.2:38235
02:03:37.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.363564 14835 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.243.2:38235 every 8 connection(s)
02:03:37.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.364962 14707 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-1/data/info.pb
02:03:37.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.369094 14284 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu as pid 14707
02:03:37.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.369302 14284 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-1/wal/instance
02:03:37.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.377414 14284 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu
02:03:37.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu
02:03:37.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/wal
02:03:37.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/data
02:03:37.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:03:37.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:03:37.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:03:37.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:03:37.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:03:37.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:03:37.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/logs
02:03:37.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/data/info.pb
02:03:37.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:03:37.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:03:37.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:03:37.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:03:37.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:03:37.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:03:37.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:03:37.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:03:37.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:03:37.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:03:37.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.13.243.3:0
02:03:37.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.13.243.3
02:03:37.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.13.243.3
02:03:37.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:03:37.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.13.243.62:45883,127.13.243.61:44961,127.13.243.60:35015
02:03:37.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.13.243.20:34675
02:03:37.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:03:37.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:03:37.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:03:37.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.387359 14838 heartbeater.cc:344] Connected to a master server at 127.13.243.61:44961
02:03:37.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.387797 14838 heartbeater.cc:461] Registering TS with master...
02:03:37.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.388973 14838 heartbeater.cc:507] Master 127.13.243.61:44961 requested a full tablet report, sending...
02:03:37.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.390965 14397 ts_manager.cc:194] Registered new tserver with Master: b123247401d540528f8324235e936260 (127.13.243.2:38235)
02:03:37.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.392789 14836 heartbeater.cc:344] Connected to a master server at 127.13.243.60:35015
02:03:37.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.392931 14836 heartbeater.cc:461] Registering TS with master...
02:03:37.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.393415 14837 heartbeater.cc:344] Connected to a master server at 127.13.243.62:45883
02:03:37.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.393477 14836 heartbeater.cc:507] Master 127.13.243.60:35015 requested a full tablet report, sending...
02:03:37.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.393577 14837 heartbeater.cc:461] Registering TS with master...
02:03:37.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.394187 14837 heartbeater.cc:507] Master 127.13.243.62:45883 requested a full tablet report, sending...
02:03:37.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.394987 14468 ts_manager.cc:194] Registered new tserver with Master: b123247401d540528f8324235e936260 (127.13.243.2:38235)
02:03:37.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.395320 14326 ts_manager.cc:194] Registered new tserver with Master: b123247401d540528f8324235e936260 (127.13.243.2:38235)
02:03:37.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.396164 14468 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.13.243.2:49739
02:03:37.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.444265 14569 catalog_manager.cc:1294] Loaded cluster ID: b6b271981afd4cf29d06d58794657092
02:03:37.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.444407 14569 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 3abc49041df14c1a9406a684f33d7ed4: loading cluster ID for follower catalog manager: success
02:03:37.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.448189 14569 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 3abc49041df14c1a9406a684f33d7ed4: acquiring CA information for follower catalog manager: success
02:03:37.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.450533 14569 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 3abc49041df14c1a9406a684f33d7ed4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
02:03:37.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:37.724509 14843 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:03:37.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:37.724861 14843 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:03:37.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:37.724977 14843 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:03:37.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:37.733572 14843 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:03:37.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:37.733814 14843 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.243.3
02:03:37.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.744277 14843 tablet_server_runner.cc:78] Tablet server non-default flags:
02:03:37.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:03:37.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.13.243.20:34675
02:03:37.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:03:37.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:03:37.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/data
02:03:37.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/wal
02:03:37.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:03:37.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:03:37.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.13.243.3:0
02:03:37.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:03:37.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:03:37.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:03:37.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/data/info.pb
02:03:37.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.13.243.3
02:03:37.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:03:37.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.13.243.62:45883,127.13.243.61:44961,127.13.243.60:35015
02:03:37.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:03:37.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:03:37.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:03:37.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:03:37.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:03:37.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.13.243.3
02:03:37.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/logs
02:03:37.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:03:37.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:03:37.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:03:37.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:03:37.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:03:37.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision a3f35afc2b6dad6aa403ec58378221544a26edd0
02:03:37.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:03:37.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:40:41 UTC on bdcb31816ec0
02:03:37.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 16849
02:03:37.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
02:03:37.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.745865 14843 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:03:37.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.747725 14843 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:03:37.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:37.758538 14850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:37.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:37.759573 14849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:37.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:37.761605 14852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:37.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.761962 14843 server_base.cc:1061] running on GCE node
02:03:37.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.762627 14843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:03:37.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.763562 14843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:03:37.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.764787 14843 hybrid_clock.cc:648] HybridClock initialized: now 1783476217764740 us; error 43 us; skew 500 ppm
02:03:37.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.765151 14843 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:03:37.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.767529 14843 webserver.cc:534] Webserver started at http://127.13.243.3:43625/ using document root <none> and password file <none>
02:03:37.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.768314 14843 fs_manager.cc:362] Metadata directory not provided
02:03:37.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.768445 14843 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:03:37.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.768828 14843 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
02:03:37.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.771368 14843 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/data/instance:
02:03:37.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "7f08c11f6dc94dc3aa69febd1051f2e9"
02:03:37.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:03:37 on dist-test-slave-pjt5"
02:03:37.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.772231 14843 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/wal/instance:
02:03:37.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "7f08c11f6dc94dc3aa69febd1051f2e9"
02:03:37.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:03:37 on dist-test-slave-pjt5"
02:03:37.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.778250 14843 fs_manager.cc:696] Time spent creating directory manager: real 0.006s	user 0.006s	sys 0.000s
02:03:37.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.782143 14858 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:03:37.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.783639 14843 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.002s
02:03:37.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.783833 14843 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/data,/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/wal
02:03:37.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "7f08c11f6dc94dc3aa69febd1051f2e9"
02:03:37.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:03:37 on dist-test-slave-pjt5"
02:03:37.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.784089 14843 fs_report.cc:389] FS layout report
02:03:37.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:03:37.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/wal
02:03:37.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/wal
02:03:37.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/data/data
02:03:37.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:03:37.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:03:37.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:03:37.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:03:37.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:03:37.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:03:37.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:03:37.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:03:37.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:03:37.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:03:37.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:03:37.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.803838 14843 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:03:37.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.804802 14843 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:03:37.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.805120 14843 kserver.cc:163] Server-wide thread pool size limit: 3276
02:03:37.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.806253 14843 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:03:37.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.808043 14843 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
02:03:37.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.808168 14843 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
02:03:37.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.808272 14843 ts_tablet_manager.cc:616] Registered 0 tablets
02:03:37.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.808349 14843 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
02:03:37.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.847396 14843 rpc_server.cc:307] RPC server started. Bound to: 127.13.243.3:41985
02:03:37.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.847451 14970 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.243.3:41985 every 8 connection(s)
02:03:37.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.848919 14843 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/data/info.pb
02:03:37.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.851239 14284 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu as pid 14843
02:03:37.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.851416 14284 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/wal/instance
02:03:37.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.876129 14972 heartbeater.cc:344] Connected to a master server at 127.13.243.62:45883
02:03:37.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.876205 14971 heartbeater.cc:344] Connected to a master server at 127.13.243.60:35015
02:03:37.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.876571 14972 heartbeater.cc:461] Registering TS with master...
02:03:37.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.876623 14973 heartbeater.cc:344] Connected to a master server at 127.13.243.61:44961
02:03:37.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.876742 14973 heartbeater.cc:461] Registering TS with master...
02:03:37.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.876569 14971 heartbeater.cc:461] Registering TS with master...
02:03:37.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.877847 14972 heartbeater.cc:507] Master 127.13.243.62:45883 requested a full tablet report, sending...
02:03:37.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.877879 14973 heartbeater.cc:507] Master 127.13.243.61:44961 requested a full tablet report, sending...
02:03:37.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.878175 14971 heartbeater.cc:507] Master 127.13.243.60:35015 requested a full tablet report, sending...
02:03:37.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.879766 14468 ts_manager.cc:194] Registered new tserver with Master: 7f08c11f6dc94dc3aa69febd1051f2e9 (127.13.243.3:41985)
02:03:37.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.879825 14325 ts_manager.cc:194] Registered new tserver with Master: 7f08c11f6dc94dc3aa69febd1051f2e9 (127.13.243.3:41985)
02:03:37.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.880806 14468 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.13.243.3:49897
02:03:37.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.881846 14396 ts_manager.cc:194] Registered new tserver with Master: 7f08c11f6dc94dc3aa69febd1051f2e9 (127.13.243.3:41985)
02:03:37.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.895707 14284 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
02:03:37.897 [DEBUG - main] (MiniKuduCluster.java:195) Response: 
02:03:37.899 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_masters {
}

02:03:37.906 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_masters {
  masters {
    id {
      type: MASTER
      index: 0
    }
    bound_rpc_address {
      host: "127.13.243.62"
      port: 45883
    }
    bound_http_address {
      host: "127.13.243.62"
      port: 40181
    }
  }
  masters {
    id {
      type: MASTER
      index: 1
    }
    bound_rpc_address {
      host: "127.13.243.61"
      port: 44961
    }
    bound_http_address {
      host: "127.13.243.61"
      port: 33995
    }
  }
  masters {
    id {
      type: MASTER
      index: 2
    }
    bound_rpc_address {
      host: "127.13.243.60"
      port: 35015
    }
    bound_http_address {
      host: "127.13.243.60"
      port: 39635
    }
  }
}

02:03:37.910 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_tservers {
}

02:03:37.913 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_tservers {
  tservers {
    id {
      type: TSERVER
      index: 0
    }
    bound_rpc_address {
      host: "127.13.243.1"
      port: 33421
    }
    bound_http_address {
      host: "127.13.243.1"
      port: 42315
    }
  }
  tservers {
    id {
      type: TSERVER
      index: 1
    }
    bound_rpc_address {
      host: "127.13.243.2"
      port: 38235
    }
    bound_http_address {
      host: "127.13.243.2"
      port: 42383
    }
  }
  tservers {
    id {
      type: TSERVER
      index: 2
    }
    bound_rpc_address {
      host: "127.13.243.3"
      port: 41985
    }
    bound_http_address {
      host: "127.13.243.3"
      port: 43625
    }
  }
}

02:03:37.913 [INFO - main] (KuduTestHarness.java:157) Creating a new Kudu client...
02:03:37.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:37.935283 14699 heartbeater.cc:499] Master 127.13.243.60:35015 was elected leader, sending a full tablet report...
02:03:38.109 [DEBUG - main] (CompositeStrategy.java:59) Unable to load modern Subject methods, relying only on legacy methods
java.lang.NoSuchMethodException: javax.security.auth.Subject.current()
	at java.base/java.lang.Class.getDeclaredMethod(Class.java:2677)
	at org.apache.kudu.client.internals.ModernStrategy.<init>(ModernStrategy.java:51)
	at org.apache.kudu.client.internals.CompositeStrategy.<init>(CompositeStrategy.java:52)
	at org.apache.kudu.client.internals.CompositeStrategy.<clinit>(CompositeStrategy.java:41)
	at org.apache.kudu.client.internals.SecurityManagerCompatibility.get(SecurityManagerCompatibility.java:43)
	at org.apache.kudu.client.SecurityContext.setupSubject(SecurityContext.java:155)
	at org.apache.kudu.client.SecurityContext.<init>(SecurityContext.java:138)
	at org.apache.kudu.client.AsyncKuduClient.<init>(AsyncKuduClient.java:431)
	at org.apache.kudu.client.AsyncKuduClient$AsyncKuduClientBuilder.build(AsyncKuduClient.java:3109)
	at org.apache.kudu.test.KuduTestHarness.before(KuduTestHarness.java:161)
	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:50)
	at org.apache.kudu.test.junit.RetryRule$RetryStatement.doOneAttempt(RetryRule.java:225)
	at org.apache.kudu.test.junit.RetryRule$RetryStatement.evaluate(RetryRule.java:246)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.junit.runners.Suite.runChild(Suite.java:128)
	at org.junit.runners.Suite.runChild(Suite.java:27)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
	at org.junit.runner.JUnitCore.runMain(JUnitCore.java:77)
	at org.junit.runner.JUnitCore.main(JUnitCore.java:36)
02:03:38.168 [DEBUG - main] (SecurityUtil.java:122) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 
02:03:38.346 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.13.243.62:45883(127.13.243.62:45883)] connecting to peer
02:03:38.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:38.398957 14836 heartbeater.cc:499] Master 127.13.243.60:35015 was elected leader, sending a full tablet report...
02:03:38.406 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.13.243.60:35015(127.13.243.60:35015)] connecting to peer
02:03:38.408 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.13.243.61:44961(127.13.243.61:44961)] connecting to peer
02:03:38.425 [DEBUG - kudu-nio-2] (Connection.java:757) [peer master-127.13.243.61:44961(127.13.243.61:44961)] Successfully connected to peer
02:03:38.429 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.13.243.62:45883(127.13.243.62:45883)] Successfully connected to peer
02:03:38.430 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.13.243.60:35015(127.13.243.60:35015)] Successfully connected to peer
02:03:38.529 [DEBUG - kudu-nio-0] (Negotiator.java:529) SASL mechanism PLAIN chosen for peer 127.13.243.62
02:03:38.530 [DEBUG - kudu-nio-2] (Negotiator.java:529) SASL mechanism PLAIN chosen for peer 127.13.243.61
02:03:38.531 [DEBUG - kudu-nio-1] (Negotiator.java:529) SASL mechanism PLAIN chosen for peer 127.13.243.60
02:03:38.737 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0x2cb70358, L:/127.0.0.1:37490 - R:/127.13.243.62:45883] using SASL/PLAIN
02:03:38.737 [DEBUG - kudu-nio-2] (Negotiator.java:919) Authenticated connection [id: 0xf75b5d27, L:/127.0.0.1:42888 - R:/127.13.243.61:44961] using SASL/PLAIN
02:03:38.737 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x419b7c5e, L:/127.0.0.1:37588 - R:/127.13.243.60:35015] using SASL/PLAIN
02:03:38.772 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet Kudu Master for table 'Kudu Master' with partition [<start>, <end>)
02:03:38.773 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=CreateTable, tablet=null, attempt=1, TimeoutTracker(timeout=50000, elapsed=470), Traces: [0ms] refreshing cache from master, [28ms] Sub RPC ConnectToMaster: sending RPC to server master-127.13.243.62:45883, [100ms] Sub RPC ConnectToMaster: sending RPC to server master-127.13.243.60:35015, [102ms] Sub RPC ConnectToMaster: sending RPC to server master-127.13.243.61:44961, [445ms] Sub RPC ConnectToMaster: received response from server master-127.13.243.61:44961: OK, [445ms] Sub RPC ConnectToMaster: received response from server master-127.13.243.62:45883: OK, [445ms] Sub RPC ConnectToMaster: received response from server master-127.13.243.60:35015: OK, deferred=null) after lookup
02:03:38.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:38.817617 14468 catalog_manager.cc:2282] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37588:
02:03:38.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) name: "org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291"
02:03:38.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) schema {
02:03:38.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
02:03:38.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "key"
02:03:38.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
02:03:38.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: true
02:03:38.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
02:03:38.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
02:03:38.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
02:03:38.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
02:03:38.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
02:03:38.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
02:03:38.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column1_i"
02:03:38.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
02:03:38.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
02:03:38.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
02:03:38.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
02:03:38.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
02:03:38.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
02:03:38.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
02:03:38.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
02:03:38.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column2_i"
02:03:38.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
02:03:38.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
02:03:38.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
02:03:38.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
02:03:38.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
02:03:38.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
02:03:38.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
02:03:38.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
02:03:38.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column3_s"
02:03:38.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: STRING
02:03:38.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
02:03:38.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: true
02:03:38.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     encoding: DICT_ENCODING
02:03:38.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     compression: LZ4
02:03:38.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 4096
02:03:38.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
02:03:38.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
02:03:38.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
02:03:38.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
02:03:38.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column4_b"
02:03:38.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: BOOL
02:03:38.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
02:03:38.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
02:03:38.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
02:03:38.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
02:03:38.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
02:03:38.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
02:03:38.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) }
02:03:38.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) partition_schema {
02:03:38.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   range_schema {
02:03:38.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     columns {
02:03:38.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)       name: "key"
02:03:38.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     }
02:03:38.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
02:03:38.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) }
02:03:38.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:38.819702 14468 catalog_manager.cc:7076] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
02:03:38.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:38.877086 14771 tablet_service.cc:1511] Processing CreateTablet for tablet 89c23a195e1e46c19d1cff4420b1f6e5 (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291 [id=99712037bd23492facd17dd88f9aec6c]), partition=RANGE (key) PARTITION UNBOUNDED
02:03:38.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:38.879072 14771 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89c23a195e1e46c19d1cff4420b1f6e5. 1 dirs total, 0 dirs full, 0 dirs failed
02:03:38.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:38.877269 14906 tablet_service.cc:1511] Processing CreateTablet for tablet 89c23a195e1e46c19d1cff4420b1f6e5 (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291 [id=99712037bd23492facd17dd88f9aec6c]), partition=RANGE (key) PARTITION UNBOUNDED
02:03:38.883 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:894) Opened table 99712037bd23492facd17dd88f9aec6c
02:03:38.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:38.883095 14971 heartbeater.cc:499] Master 127.13.243.60:35015 was elected leader, sending a full tablet report...
02:03:38.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:38.885957 14906 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89c23a195e1e46c19d1cff4420b1f6e5. 1 dirs total, 0 dirs full, 0 dirs failed
02:03:38.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:38.901350 14634 tablet_service.cc:1511] Processing CreateTablet for tablet 89c23a195e1e46c19d1cff4420b1f6e5 (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291 [id=99712037bd23492facd17dd88f9aec6c]), partition=RANGE (key) PARTITION UNBOUNDED
02:03:38.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:38.902046 15003 tablet_bootstrap.cc:492] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260: Bootstrap starting.
02:03:38.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:38.903481 14634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89c23a195e1e46c19d1cff4420b1f6e5. 1 dirs total, 0 dirs full, 0 dirs failed
02:03:38.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:38.903895 15004 tablet_bootstrap.cc:492] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9: Bootstrap starting.
02:03:38.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:38.905793 15003 tablet_bootstrap.cc:654] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260: Neither blocks nor log segments found. Creating new log.
02:03:38.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:38.907241 15003 log.cc:826] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260: Log is configured to *not* fsync() on all Append() calls
02:03:38.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:38.907557 15004 tablet_bootstrap.cc:654] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9: Neither blocks nor log segments found. Creating new log.
02:03:38.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:38.909235 15004 log.cc:826] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9: Log is configured to *not* fsync() on all Append() calls
02:03:38.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:38.917660 15003 tablet_bootstrap.cc:492] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260: No bootstrap required, opened a new log
02:03:38.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:38.918197 15003 ts_tablet_manager.cc:1403] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260: Time spent bootstrapping tablet: real 0.016s	user 0.007s	sys 0.007s
02:03:38.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:38.919709 15004 tablet_bootstrap.cc:492] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9: No bootstrap required, opened a new log
02:03:38.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:38.920148 15004 ts_tablet_manager.cc:1403] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9: Time spent bootstrapping tablet: real 0.016s	user 0.007s	sys 0.007s
02:03:38.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:38.925791 15003 raft_consensus.cc:359] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:03:38.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:38.926484 15003 raft_consensus.cc:385] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
02:03:38.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:38.926729 15003 raft_consensus.cc:740] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b123247401d540528f8324235e936260, State: Initialized, Role: FOLLOWER
02:03:38.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:38.927623 15003 consensus_queue.cc:260] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:03:38.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:38.928440 15004 raft_consensus.cc:359] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:03:38.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:38.928887 15007 tablet_bootstrap.cc:492] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101: Bootstrap starting.
02:03:38.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:38.928972 15004 raft_consensus.cc:385] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
02:03:38.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:38.929075 15004 raft_consensus.cc:740] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f08c11f6dc94dc3aa69febd1051f2e9, State: Initialized, Role: FOLLOWER
02:03:38.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:38.929740 15004 consensus_queue.cc:260] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:03:38.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:38.932126 15003 ts_tablet_manager.cc:1434] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260: Time spent starting tablet: real 0.014s	user 0.008s	sys 0.002s
02:03:38.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:38.941761 15007 tablet_bootstrap.cc:654] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101: Neither blocks nor log segments found. Creating new log.
02:03:38.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:38.943338 15007 log.cc:826] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101: Log is configured to *not* fsync() on all Append() calls
02:03:38.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:38.951660 15009 raft_consensus.cc:493] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
02:03:38.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:38.951967 15009 raft_consensus.cc:515] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:03:38.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:38.953249 15004 ts_tablet_manager.cc:1434] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9: Time spent starting tablet: real 0.033s	user 0.002s	sys 0.013s
02:03:38.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:38.959713 15007 tablet_bootstrap.cc:492] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101: No bootstrap required, opened a new log
02:03:38.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:38.960376 15007 ts_tablet_manager.cc:1403] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101: Time spent bootstrapping tablet: real 0.032s	user 0.015s	sys 0.000s
02:03:38.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:38.961062 15009 leader_election.cc:290] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6422ba2afbf742febb59b258c6ac5101 (127.13.243.1:33421), b123247401d540528f8324235e936260 (127.13.243.2:38235)
02:03:38.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:38.968840 15007 raft_consensus.cc:359] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:03:38.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:38.969269 15007 raft_consensus.cc:385] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
02:03:38.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:38.969372 15007 raft_consensus.cc:740] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6422ba2afbf742febb59b258c6ac5101, State: Initialized, Role: FOLLOWER
02:03:38.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:38.970150 15007 consensus_queue.cc:260] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:03:38.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:38.970515 14791 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89c23a195e1e46c19d1cff4420b1f6e5" candidate_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b123247401d540528f8324235e936260" is_pre_election: true
02:03:38.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:38.971092 14791 raft_consensus.cc:2468] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f08c11f6dc94dc3aa69febd1051f2e9 in term 0.
02:03:38.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:38.972005 14861 leader_election.cc:304] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f08c11f6dc94dc3aa69febd1051f2e9, b123247401d540528f8324235e936260; no voters: 
02:03:38.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:38.972366 15007 ts_tablet_manager.cc:1434] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101: Time spent starting tablet: real 0.012s	user 0.010s	sys 0.000s
02:03:38.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:38.972694 15009 raft_consensus.cc:2804] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
02:03:38.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:38.972855 15009 raft_consensus.cc:493] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
02:03:38.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:38.972983 15009 raft_consensus.cc:3060] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [term 0 FOLLOWER]: Advancing to term 1
02:03:38.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:38.976163 15009 raft_consensus.cc:515] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:03:38.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:38.976819 15009 leader_election.cc:290] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [CANDIDATE]: Term 1 election: Requested vote from peers 6422ba2afbf742febb59b258c6ac5101 (127.13.243.1:33421), b123247401d540528f8324235e936260 (127.13.243.2:38235)
02:03:38.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:38.977953 14654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89c23a195e1e46c19d1cff4420b1f6e5" candidate_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6422ba2afbf742febb59b258c6ac5101" is_pre_election: true
02:03:38.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:38.978472 14654 raft_consensus.cc:2468] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f08c11f6dc94dc3aa69febd1051f2e9 in term 0.
02:03:38.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:38.979672 14654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89c23a195e1e46c19d1cff4420b1f6e5" candidate_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6422ba2afbf742febb59b258c6ac5101"
02:03:38.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:38.979842 14654 raft_consensus.cc:3060] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [term 0 FOLLOWER]: Advancing to term 1
02:03:38.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:38.981284 14791 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89c23a195e1e46c19d1cff4420b1f6e5" candidate_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b123247401d540528f8324235e936260"
02:03:38.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:38.981499 14791 raft_consensus.cc:3060] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [term 0 FOLLOWER]: Advancing to term 1
02:03:38.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:38.983006 14654 raft_consensus.cc:2468] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f08c11f6dc94dc3aa69febd1051f2e9 in term 1.
02:03:38.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:38.983423 14859 leader_election.cc:304] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6422ba2afbf742febb59b258c6ac5101, 7f08c11f6dc94dc3aa69febd1051f2e9; no voters: 
02:03:38.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:38.983734 15009 raft_consensus.cc:2804] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [term 1 FOLLOWER]: Leader election won for term 1
02:03:38.984 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:38.984201 14791 raft_consensus.cc:2468] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f08c11f6dc94dc3aa69febd1051f2e9 in term 1.
02:03:38.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:38.985198 15009 raft_consensus.cc:697] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [term 1 LEADER]: Becoming Leader. State: Replica: 7f08c11f6dc94dc3aa69febd1051f2e9, State: Running, Role: LEADER
02:03:38.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:38.985775 15009 consensus_queue.cc:237] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:03:38.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:38.991883 14468 catalog_manager.cc:5696] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f08c11f6dc94dc3aa69febd1051f2e9 (127.13.243.3). New cstate: current_term: 1 leader_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } health_report { overall_health: HEALTHY } } }
02:03:39.010 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:894) Opened table 99712037bd23492facd17dd88f9aec6c
02:03:39.031 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:03:39.037 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299994, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421),7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)[L],b123247401d540528f8324235e936260(127.13.243.2:38235)]}]
02:03:39.039 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=null, attempt=1, sequence_id=-1, TimeoutTracker(timeout=30000, elapsed=21), Traces: [0ms] refreshing cache from master, [0ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [11ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=null) after lookup
02:03:39.048 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)] connecting to peer
02:03:39.050 [DEBUG - kudu-nio-3] (Connection.java:757) [peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)] Successfully connected to peer
02:03:39.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:39.103579 14974 tablet.cc:2406] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
02:03:39.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:39.119401 14839 tablet.cc:2406] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260: Can't schedule compaction. Clean time has not been advanced past its initial value.
02:03:39.120 [DEBUG - kudu-nio-3] (Negotiator.java:919) Authenticated connection [id: 0xfe4b8b98, L:/127.0.0.1:39684 - R:/127.13.243.3:41985] using TOKEN/null
02:03:39.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:39.136756 14654 raft_consensus.cc:1275] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [term 1 FOLLOWER]: Refusing update from remote peer 7f08c11f6dc94dc3aa69febd1051f2e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
02:03:39.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:39.136756 14791 raft_consensus.cc:1275] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [term 1 FOLLOWER]: Refusing update from remote peer 7f08c11f6dc94dc3aa69febd1051f2e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
02:03:39.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:39.137935 15011 consensus_queue.cc:1048] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
02:03:39.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:39.138458 15009 consensus_queue.cc:1048] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
02:03:39.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:39.156860 15018 mvcc.cc:204] Tried to move back new op lower bound from 7305118593571819520 to 7305118592971321344. Current Snapshot: MvccSnapshot[applied={T|T < 7305118593571819520}]
02:03:39.283 [DEBUG - kudu-nio-3] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 3 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291, tablet=89c23a195e1e46c19d1cff4420b1f6e5, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:03:39.286 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.13.243.3:41985
02:03:39.287 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

02:03:39.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:39.288481 14284 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu pid 14843
02:03:39.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:39.303005 14966 generic_service.cc:196] Checking for leaks (request via RPC)
02:03:39.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:39.407805 14284 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu with pid 14843
02:03:39.429 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:03:39.433 [DEBUG - kudu-nio-3] (Connection.java:239) [peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)] handling channelInactive
02:03:39.434 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)] cleaning up while in state READY due to: connection closed
02:03:39.434 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985) for tablet 89c23a195e1e46c19d1cff4420b1f6e5: connection closed
02:03:39.447 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:03:39.448 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421),7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)[L],b123247401d540528f8324235e936260(127.13.243.2:38235)]}]
02:03:39.448 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=89c23a195e1e46c19d1cff4420b1f6e5, attempt=2, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=18), Traces: [0ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [4ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: connection closed, [4ms] delaying RPC due to: Network error: connection closed, [14ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [17ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=Deferred@705772718(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f87c036bed0@57c24f37 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1d3a7359 -> wakeup thread Time-limited test)) after lookup
02:03:39.449 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)] connecting to peer
02:03:39.451 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:03:39.451 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:03:39.451 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985) for tablet 89c23a195e1e46c19d1cff4420b1f6e5: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:03:39.467 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:03:39.468 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421),7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)[L],b123247401d540528f8324235e936260(127.13.243.2:38235)]}]
02:03:39.468 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=89c23a195e1e46c19d1cff4420b1f6e5, attempt=4, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=38), Traces: [0ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [4ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: connection closed, [4ms] delaying RPC due to: Network error: connection closed, [14ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [17ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [18ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [21ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [22ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [34ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [37ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=Deferred@705772718(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f87c036bed0@57c24f37 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@315023291 after retry RPC@554188664), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1d3a7359 -> wakeup thread Time-limited test -> (continuation of Deferred@315023291 after retry RPC@554188664))) after lookup
02:03:39.469 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)] connecting to peer
02:03:39.470 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:03:39.471 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:03:39.471 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985) for tablet 89c23a195e1e46c19d1cff4420b1f6e5: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:03:39.487 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:03:39.488 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421),7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)[L],b123247401d540528f8324235e936260(127.13.243.2:38235)]}]
02:03:39.488 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=89c23a195e1e46c19d1cff4420b1f6e5, attempt=6, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=58), Traces: [0ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [4ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: connection closed, [4ms] delaying RPC due to: Network error: connection closed, [14ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [17ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [18ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [21ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [22ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [34ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [37ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [38ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [41ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [41ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [54ms] refreshing cache from master, [54ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [57ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=Deferred@705772718(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f87c036bed0@57c24f37 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@315023291 after retry RPC@554188664) -> (continuation of Deferred@1387397130 after retry RPC@1278782135), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1d3a7359 -> wakeup thread Time-limited test -> (continuation of Deferred@315023291 after retry RPC@554188664) -> (continuation of Deferred@1387397130 after retry RPC@1278782135))) after lookup
02:03:39.489 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)] connecting to peer
02:03:39.491 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:03:39.491 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:03:39.491 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985) for tablet 89c23a195e1e46c19d1cff4420b1f6e5: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:03:39.549 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:03:39.549 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421),7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)[L],b123247401d540528f8324235e936260(127.13.243.2:38235)]}]
02:03:39.550 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=89c23a195e1e46c19d1cff4420b1f6e5, attempt=8, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=119), Traces: [0ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [4ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: connection closed, [4ms] delaying RPC due to: Network error: connection closed, [14ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [17ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [18ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [21ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [22ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [34ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [37ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [38ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [41ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [41ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [54ms] refreshing cache from master, [54ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [57ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [59ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [61ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [62ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [114ms] refreshing cache from master, [114ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [118ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=Deferred@705772718(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f87c036bed0@57c24f37 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@315023291 after retry RPC@554188664) -> (continuation of Deferred@1387397130 after retry RPC@1278782135) -> (continuation of Deferred@669001349 after retry RPC@1331858327), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1d3a7359 -> wakeup thread Time-limited test -> (continuation of Deferred@315023291 after retry RPC@554188664) -> (continuation of Deferred@1387397130 after retry RPC@1278782135) -> (continuation of Deferred@669001349 after retry RPC@1331858327))) after lookup
02:03:39.551 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)] connecting to peer
02:03:39.553 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:03:39.553 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:03:39.553 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985) for tablet 89c23a195e1e46c19d1cff4420b1f6e5: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:03:39.847 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:03:39.848 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421),7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)[L],b123247401d540528f8324235e936260(127.13.243.2:38235)]}]
02:03:39.851 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=89c23a195e1e46c19d1cff4420b1f6e5, attempt=10, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=418), Traces: [0ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [4ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: connection closed, [4ms] delaying RPC due to: Network error: connection closed, [14ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [17ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [18ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [21ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [22ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [34ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [37ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [38ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [41ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [41ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [54ms] refreshing cache from master, [54ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [57ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [59ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [61ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [62ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [114ms] refreshing cache from master, [114ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [118ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [120ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [123ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [123ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [414ms] refreshing cache from master, [414ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [417ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=Deferred@705772718(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f87c036bed0@57c24f37 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@315023291 after retry RPC@554188664) -> (continuation of Deferred@1387397130 after retry RPC@1278782135) -> (continuation of Deferred@669001349 after retry RPC@1331858327) -> (continuation of Deferred@644703374 after retry RPC@479803502), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1d3a7359 -> wakeup thread Time-limited test -> (continuation of Deferred@315023291 after retry RPC@554188664) -> (continuation of Deferred@1387397130 after retry RPC@1278782135) -> (continuation of Deferred@669001349 after retry RPC@1331858327) -> (continuation of Deferred@644703374 after retry RPC@479803502))) after lookup
02:03:39.852 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)] connecting to peer
02:03:39.853 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:03:39.853 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:03:39.853 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985) for tablet 89c23a195e1e46c19d1cff4420b1f6e5: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:03:40.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:40.719934 15035 raft_consensus.cc:493] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7f08c11f6dc94dc3aa69febd1051f2e9)
02:03:40.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:40.720180 15035 raft_consensus.cc:515] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:03:40.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:40.721839 15035 leader_election.cc:290] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6422ba2afbf742febb59b258c6ac5101 (127.13.243.1:33421), 7f08c11f6dc94dc3aa69febd1051f2e9 (127.13.243.3:41985)
02:03:40.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:40.722675 15036 raft_consensus.cc:493] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7f08c11f6dc94dc3aa69febd1051f2e9)
02:03:40.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:40.723208 15036 raft_consensus.cc:515] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:03:40.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:40.724361 14724 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.243.3:41985: connect: Connection refused (error 111)
02:03:40.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:40.725946 14724 leader_election.cc:336] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7f08c11f6dc94dc3aa69febd1051f2e9 (127.13.243.3:41985): Network error: Client connection negotiation failed: client connection to 127.13.243.3:41985: connect: Connection refused (error 111)
02:03:40.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:40.726629 15036 leader_election.cc:290] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b123247401d540528f8324235e936260 (127.13.243.2:38235), 7f08c11f6dc94dc3aa69febd1051f2e9 (127.13.243.3:41985)
02:03:40.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:40.728324 14587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.243.3:41985: connect: Connection refused (error 111)
02:03:40.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:40.730252 14587 leader_election.cc:336] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7f08c11f6dc94dc3aa69febd1051f2e9 (127.13.243.3:41985): Network error: Client connection negotiation failed: client connection to 127.13.243.3:41985: connect: Connection refused (error 111)
02:03:40.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:40.732558 14654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89c23a195e1e46c19d1cff4420b1f6e5" candidate_uuid: "b123247401d540528f8324235e936260" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6422ba2afbf742febb59b258c6ac5101" is_pre_election: true
02:03:40.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:40.732784 14654 raft_consensus.cc:2468] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b123247401d540528f8324235e936260 in term 1.
02:03:40.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:40.733349 14724 leader_election.cc:304] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6422ba2afbf742febb59b258c6ac5101, b123247401d540528f8324235e936260; no voters: 7f08c11f6dc94dc3aa69febd1051f2e9
02:03:40.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:40.733785 15035 raft_consensus.cc:2804] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [term 1 FOLLOWER]: Leader pre-election won for term 2
02:03:40.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:40.734027 15035 raft_consensus.cc:493] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7f08c11f6dc94dc3aa69febd1051f2e9)
02:03:40.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:40.734174 15035 raft_consensus.cc:3060] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [term 1 FOLLOWER]: Advancing to term 2
02:03:40.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:40.734740 14791 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89c23a195e1e46c19d1cff4420b1f6e5" candidate_uuid: "6422ba2afbf742febb59b258c6ac5101" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b123247401d540528f8324235e936260" is_pre_election: true
02:03:40.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:40.736552 15035 raft_consensus.cc:515] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:03:40.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:40.736896 14791 raft_consensus.cc:2393] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6422ba2afbf742febb59b258c6ac5101 in current term 2: Already voted for candidate b123247401d540528f8324235e936260 in this term.
02:03:40.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:40.737283 15035 leader_election.cc:290] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [CANDIDATE]: Term 2 election: Requested vote from peers 6422ba2afbf742febb59b258c6ac5101 (127.13.243.1:33421), 7f08c11f6dc94dc3aa69febd1051f2e9 (127.13.243.3:41985)
02:03:40.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:40.737675 14654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89c23a195e1e46c19d1cff4420b1f6e5" candidate_uuid: "b123247401d540528f8324235e936260" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6422ba2afbf742febb59b258c6ac5101"
02:03:40.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:40.737874 14654 raft_consensus.cc:3060] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [term 1 FOLLOWER]: Advancing to term 2
02:03:40.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:40.737946 14589 leader_election.cc:304] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6422ba2afbf742febb59b258c6ac5101; no voters: 7f08c11f6dc94dc3aa69febd1051f2e9, b123247401d540528f8324235e936260
02:03:40.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:40.739287 14724 leader_election.cc:336] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 7f08c11f6dc94dc3aa69febd1051f2e9 (127.13.243.3:41985): Network error: Client connection negotiation failed: client connection to 127.13.243.3:41985: connect: Connection refused (error 111)
02:03:40.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:40.740127 14654 raft_consensus.cc:2468] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b123247401d540528f8324235e936260 in term 2.
02:03:40.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:40.740336 15036 raft_consensus.cc:2749] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
02:03:40.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:40.740738 14724 leader_election.cc:304] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6422ba2afbf742febb59b258c6ac5101, b123247401d540528f8324235e936260; no voters: 7f08c11f6dc94dc3aa69febd1051f2e9
02:03:40.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:40.741037 15035 raft_consensus.cc:2804] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [term 2 FOLLOWER]: Leader election won for term 2
02:03:40.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:40.741412 15035 raft_consensus.cc:697] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [term 2 LEADER]: Becoming Leader. State: Replica: b123247401d540528f8324235e936260, State: Running, Role: LEADER
02:03:40.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:40.741955 15035 consensus_queue.cc:237] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:03:40.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:40.745502 14467 catalog_manager.cc:5696] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 reported cstate change: term changed from 1 to 2, leader changed from 7f08c11f6dc94dc3aa69febd1051f2e9 (127.13.243.3) to b123247401d540528f8324235e936260 (127.13.243.2). New cstate: current_term: 2 leader_uuid: "b123247401d540528f8324235e936260" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } health_report { overall_health: UNKNOWN } } }
02:03:41.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:41.207501 14724 consensus_peers.cc:597] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 -> Peer 7f08c11f6dc94dc3aa69febd1051f2e9 (127.13.243.3:41985): Couldn't send request to peer 7f08c11f6dc94dc3aa69febd1051f2e9. Status: Network error: Client connection negotiation failed: client connection to 127.13.243.3:41985: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
02:03:41.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:41.247072 14654 raft_consensus.cc:1275] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [term 2 FOLLOWER]: Refusing update from remote peer b123247401d540528f8324235e936260: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
02:03:41.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:41.247797 15035 consensus_queue.cc:1048] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
02:03:41.328 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:03:41.328 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421),7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985),b123247401d540528f8324235e936260(127.13.243.2:38235)[L]]}]
02:03:41.329 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=89c23a195e1e46c19d1cff4420b1f6e5, attempt=12, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=1898), Traces: [0ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [4ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: connection closed, [4ms] delaying RPC due to: Network error: connection closed, [14ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [17ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [18ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [21ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [22ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [34ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [37ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [38ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [41ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [41ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [54ms] refreshing cache from master, [54ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [57ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [59ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [61ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [62ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [114ms] refreshing cache from master, [114ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [118ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [120ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [123ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [123ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [414ms] refreshing cache from master, [414ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [417ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [421ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [423ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [423ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [1894ms] refreshing cache from master, [1894ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [1897ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=Deferred@705772718(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f87c036bed0@57c24f37 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@315023291 after retry RPC@554188664) -> (continuation of Deferred@1387397130 after retry RPC@1278782135) -> (continuation of Deferred@669001349 after retry RPC@1331858327) -> (continuation of Deferred@644703374 after retry RPC@479803502) -> (continuation of Deferred@1632693554 after retry RPC@878699326), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1d3a7359 -> wakeup thread Time-limited test -> (continuation of Deferred@315023291 after retry RPC@554188664) -> (continuation of Deferred@1387397130 after retry RPC@1278782135) -> (continuation of Deferred@669001349 after retry RPC@1331858327) -> (continuation of Deferred@644703374 after retry RPC@479803502) -> (continuation of Deferred@1632693554 after retry RPC@878699326))) after lookup
02:03:41.330 [DEBUG - kudu-nio-1] (Connection.java:748) [peer b123247401d540528f8324235e936260(127.13.243.2:38235)] connecting to peer
02:03:41.331 [DEBUG - kudu-nio-1] (Connection.java:757) [peer b123247401d540528f8324235e936260(127.13.243.2:38235)] Successfully connected to peer
02:03:41.352 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x9d5b9dc3, L:/127.0.0.1:55896 - R:/127.13.243.2:38235] using TOKEN/null
02:03:41.379 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.13.243.3:41985
02:03:41.380 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

02:03:41.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:41.381778 14284 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu
02:03:41.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu
02:03:41.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/wal
02:03:41.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/data
02:03:41.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:03:41.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:03:41.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:03:41.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:03:41.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:03:41.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:03:41.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/logs
02:03:41.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/data/info.pb
02:03:41.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:03:41.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:03:41.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:03:41.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:03:41.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:03:41.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:03:41.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:03:41.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:03:41.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:03:41.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:03:41.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.13.243.3:41985
02:03:41.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.13.243.3
02:03:41.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.13.243.62:45883,127.13.243.61:44961,127.13.243.60:35015
02:03:41.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=43625
02:03:41.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.13.243.3
02:03:41.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.13.243.20:34675
02:03:41.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:03:41.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:03:41.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:03:41.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:41.706748 15068 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:03:41.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:41.707058 15068 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:03:41.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:41.707142 15068 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:03:41.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:41.715440 15068 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:03:41.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:41.715615 15068 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.243.3
02:03:41.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:41.725577 15068 tablet_server_runner.cc:78] Tablet server non-default flags:
02:03:41.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:03:41.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.13.243.20:34675
02:03:41.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:03:41.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:03:41.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/data
02:03:41.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/wal
02:03:41.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:03:41.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:03:41.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.13.243.3:41985
02:03:41.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:03:41.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:03:41.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:03:41.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/data/info.pb
02:03:41.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.13.243.3
02:03:41.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=43625
02:03:41.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.13.243.62:45883,127.13.243.61:44961,127.13.243.60:35015
02:03:41.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:03:41.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:03:41.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:03:41.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:03:41.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:03:41.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.13.243.3
02:03:41.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/logs
02:03:41.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:03:41.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:03:41.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:03:41.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:03:41.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:03:41.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision a3f35afc2b6dad6aa403ec58378221544a26edd0
02:03:41.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:03:41.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:40:41 UTC on bdcb31816ec0
02:03:41.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 16849
02:03:41.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
02:03:41.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:41.727324 15068 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:03:41.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:41.729040 15068 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:03:41.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:41.738869 15073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:41.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:41.738912 15074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:41.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:41.739833 15076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:41.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:41.740115 15068 server_base.cc:1061] running on GCE node
02:03:41.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:41.740746 15068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:03:41.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:41.741730 15068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:03:41.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:41.742964 15068 hybrid_clock.cc:648] HybridClock initialized: now 1783476221742921 us; error 31 us; skew 500 ppm
02:03:41.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:41.743320 15068 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:03:41.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:41.745707 15068 webserver.cc:534] Webserver started at http://127.13.243.3:43625/ using document root <none> and password file <none>
02:03:41.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:41.746556 15068 fs_manager.cc:362] Metadata directory not provided
02:03:41.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:41.746708 15068 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:03:41.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:41.752019 15068 fs_manager.cc:714] Time spent opening directory manager: real 0.003s	user 0.005s	sys 0.000s
02:03:41.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:41.755089 15084 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:03:41.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:41.756484 15068 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
02:03:41.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:41.756632 15068 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/data,/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/wal
02:03:41.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "7f08c11f6dc94dc3aa69febd1051f2e9"
02:03:41.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:03:37 on dist-test-slave-pjt5"
02:03:41.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:41.757308 15068 fs_report.cc:389] FS layout report
02:03:41.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:03:41.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/wal
02:03:41.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/wal
02:03:41.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/data/data
02:03:41.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:03:41.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:03:41.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:03:41.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:03:41.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:03:41.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:03:41.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:03:41.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:03:41.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:03:41.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:03:41.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:03:41.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:41.784420 15068 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:03:41.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:41.785272 15068 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:03:41.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:41.785527 15068 kserver.cc:163] Server-wide thread pool size limit: 3276
02:03:41.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:41.786499 15068 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:03:41.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:41.788568 15091 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
02:03:41.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:41.794535 15068 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
02:03:41.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:41.794627 15068 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.006s	user 0.000s	sys 0.001s
02:03:41.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:41.794729 15068 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
02:03:41.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:41.797798 15068 ts_tablet_manager.cc:616] Registered 1 tablets
02:03:41.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:41.797896 15068 ts_tablet_manager.cc:595] Time spent register tablets: real 0.003s	user 0.000s	sys 0.000s
02:03:41.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:41.798418 15091 tablet_bootstrap.cc:492] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9: Bootstrap starting.
02:03:41.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:41.820516 15091 log.cc:826] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9: Log is configured to *not* fsync() on all Append() calls
02:03:41.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:41.835786 15091 tablet_bootstrap.cc:492] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
02:03:41.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:41.836428 15091 tablet_bootstrap.cc:492] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9: Bootstrap complete.
02:03:41.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:41.837606 15091 ts_tablet_manager.cc:1403] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9: Time spent bootstrapping tablet: real 0.040s	user 0.029s	sys 0.007s
02:03:41.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:41.844547 15091 raft_consensus.cc:359] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:03:41.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:41.845310 15068 rpc_server.cc:307] RPC server started. Bound to: 127.13.243.3:41985
02:03:41.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:41.845374 15091 raft_consensus.cc:740] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f08c11f6dc94dc3aa69febd1051f2e9, State: Initialized, Role: FOLLOWER
02:03:41.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:41.845397 15198 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.243.3:41985 every 8 connection(s)
02:03:41.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:41.846239 15091 consensus_queue.cc:260] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:03:41.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:41.846925 15068 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/data/info.pb
02:03:41.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:41.848446 15091 ts_tablet_manager.cc:1434] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9: Time spent starting tablet: real 0.011s	user 0.013s	sys 0.000s
02:03:41.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:41.848845 14284 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu as pid 15068
02:03:41.849 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:03:41.884 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 6 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291, tablet=89c23a195e1e46c19d1cff4420b1f6e5, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:03:41.885 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.13.243.2:38235
02:03:41.885 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

02:03:41.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:41.885836 14284 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu pid 14707
02:03:41.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:41.899293 15200 heartbeater.cc:344] Connected to a master server at 127.13.243.60:35015
02:03:41.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:41.899677 15200 heartbeater.cc:461] Registering TS with master...
02:03:41.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:41.900723 15200 heartbeater.cc:507] Master 127.13.243.60:35015 requested a full tablet report, sending...
02:03:41.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:41.902956 15202 heartbeater.cc:344] Connected to a master server at 127.13.243.61:44961
02:03:41.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:41.903093 15202 heartbeater.cc:461] Registering TS with master...
02:03:41.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:41.903560 15202 heartbeater.cc:507] Master 127.13.243.61:44961 requested a full tablet report, sending...
02:03:41.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:41.903729 14467 ts_manager.cc:194] Re-registered known tserver with Master: 7f08c11f6dc94dc3aa69febd1051f2e9 (127.13.243.3:41985)
02:03:41.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:41.904963 14396 ts_manager.cc:194] Re-registered known tserver with Master: 7f08c11f6dc94dc3aa69febd1051f2e9 (127.13.243.3:41985)
02:03:41.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:41.905304 14467 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.13.243.3:58255
02:03:41.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:41.908059 15200 heartbeater.cc:499] Master 127.13.243.60:35015 was elected leader, sending a full tablet report...
02:03:41.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:41.909929 15201 heartbeater.cc:344] Connected to a master server at 127.13.243.62:45883
02:03:41.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:41.910085 15201 heartbeater.cc:461] Registering TS with master...
02:03:41.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:41.910423 15201 heartbeater.cc:507] Master 127.13.243.62:45883 requested a full tablet report, sending...
02:03:41.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:41.910928 14831 generic_service.cc:196] Checking for leaks (request via RPC)
02:03:41.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:41.911913 14326 ts_manager.cc:194] Re-registered known tserver with Master: 7f08c11f6dc94dc3aa69febd1051f2e9 (127.13.243.3:41985)
02:03:42.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:42.006510 14284 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu with pid 14707
02:03:42.025 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:03:42.025 [DEBUG - kudu-nio-1] (Connection.java:239) [peer b123247401d540528f8324235e936260(127.13.243.2:38235)] handling channelInactive
02:03:42.025 [DEBUG - kudu-nio-1] (Connection.java:709) [peer b123247401d540528f8324235e936260(127.13.243.2:38235)] cleaning up while in state READY due to: connection closed
02:03:42.044 [DEBUG - pool-2-thread-1] (Connection.java:748) [peer b123247401d540528f8324235e936260(127.13.243.2:38235)] connecting to peer
02:03:42.045 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235
02:03:42.045 [DEBUG - kudu-nio-2] (Connection.java:709) [peer b123247401d540528f8324235e936260(127.13.243.2:38235)] cleaning up while in state CONNECTING due to: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235
02:03:42.046 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location b123247401d540528f8324235e936260(127.13.243.2:38235) for tablet 89c23a195e1e46c19d1cff4420b1f6e5: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235
02:03:42.067 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:03:42.068 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421),7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985),b123247401d540528f8324235e936260(127.13.243.2:38235)[L]]}]
02:03:42.068 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=89c23a195e1e46c19d1cff4420b1f6e5, attempt=3, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=42), Traces: [0ms] sending RPC to server b123247401d540528f8324235e936260, [0ms] delaying RPC due to: Illegal state: connection is terminated, [19ms] sending RPC to server b123247401d540528f8324235e936260, [21ms] received response from server b123247401d540528f8324235e936260: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [21ms] delaying RPC due to: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [42ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=Deferred@472831192(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f87c036bed0@3e6d67cf -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3dd39632 -> wakeup thread Time-limited test)) after lookup
02:03:42.068 [DEBUG - kudu-nio-1] (Connection.java:748) [peer b123247401d540528f8324235e936260(127.13.243.2:38235)] connecting to peer
02:03:42.069 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235
02:03:42.069 [DEBUG - kudu-nio-3] (Connection.java:709) [peer b123247401d540528f8324235e936260(127.13.243.2:38235)] cleaning up while in state CONNECTING due to: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235
02:03:42.070 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location b123247401d540528f8324235e936260(127.13.243.2:38235) for tablet 89c23a195e1e46c19d1cff4420b1f6e5: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235
02:03:42.086 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:03:42.087 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421),7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985),b123247401d540528f8324235e936260(127.13.243.2:38235)[L]]}]
02:03:42.087 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=89c23a195e1e46c19d1cff4420b1f6e5, attempt=5, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=61), Traces: [0ms] sending RPC to server b123247401d540528f8324235e936260, [0ms] delaying RPC due to: Illegal state: connection is terminated, [19ms] sending RPC to server b123247401d540528f8324235e936260, [21ms] received response from server b123247401d540528f8324235e936260: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [21ms] delaying RPC due to: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [42ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [43ms] sending RPC to server b123247401d540528f8324235e936260, [44ms] received response from server b123247401d540528f8324235e936260: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [45ms] delaying RPC due to: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [58ms] refreshing cache from master, [59ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [61ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=Deferred@472831192(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f87c036bed0@3e6d67cf -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@921948274 after retry RPC@397456024), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3dd39632 -> wakeup thread Time-limited test -> (continuation of Deferred@921948274 after retry RPC@397456024))) after lookup
02:03:42.087 [DEBUG - kudu-nio-1] (Connection.java:748) [peer b123247401d540528f8324235e936260(127.13.243.2:38235)] connecting to peer
02:03:42.088 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235
02:03:42.088 [DEBUG - kudu-nio-4] (Connection.java:709) [peer b123247401d540528f8324235e936260(127.13.243.2:38235)] cleaning up while in state CONNECTING due to: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235
02:03:42.089 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location b123247401d540528f8324235e936260(127.13.243.2:38235) for tablet 89c23a195e1e46c19d1cff4420b1f6e5: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235
02:03:42.167 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:03:42.167 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421),7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985),b123247401d540528f8324235e936260(127.13.243.2:38235)[L]]}]
02:03:42.167 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=89c23a195e1e46c19d1cff4420b1f6e5, attempt=7, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=142), Traces: [0ms] sending RPC to server b123247401d540528f8324235e936260, [0ms] delaying RPC due to: Illegal state: connection is terminated, [19ms] sending RPC to server b123247401d540528f8324235e936260, [21ms] received response from server b123247401d540528f8324235e936260: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [21ms] delaying RPC due to: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [42ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [43ms] sending RPC to server b123247401d540528f8324235e936260, [44ms] received response from server b123247401d540528f8324235e936260: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [45ms] delaying RPC due to: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [58ms] refreshing cache from master, [59ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [61ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [62ms] sending RPC to server b123247401d540528f8324235e936260, [64ms] received response from server b123247401d540528f8324235e936260: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [64ms] delaying RPC due to: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [138ms] refreshing cache from master, [139ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [141ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=Deferred@472831192(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f87c036bed0@3e6d67cf -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@921948274 after retry RPC@397456024) -> (continuation of Deferred@1092348344 after retry RPC@1403893617), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3dd39632 -> wakeup thread Time-limited test -> (continuation of Deferred@921948274 after retry RPC@397456024) -> (continuation of Deferred@1092348344 after retry RPC@1403893617))) after lookup
02:03:42.168 [DEBUG - kudu-nio-1] (Connection.java:748) [peer b123247401d540528f8324235e936260(127.13.243.2:38235)] connecting to peer
02:03:42.169 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235
02:03:42.169 [DEBUG - kudu-nio-5] (Connection.java:709) [peer b123247401d540528f8324235e936260(127.13.243.2:38235)] cleaning up while in state CONNECTING due to: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235
02:03:42.169 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location b123247401d540528f8324235e936260(127.13.243.2:38235) for tablet 89c23a195e1e46c19d1cff4420b1f6e5: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235
02:03:42.187 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:03:42.187 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421),7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985),b123247401d540528f8324235e936260(127.13.243.2:38235)[L]]}]
02:03:42.188 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=89c23a195e1e46c19d1cff4420b1f6e5, attempt=9, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=162), Traces: [0ms] sending RPC to server b123247401d540528f8324235e936260, [0ms] delaying RPC due to: Illegal state: connection is terminated, [19ms] sending RPC to server b123247401d540528f8324235e936260, [21ms] received response from server b123247401d540528f8324235e936260: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [21ms] delaying RPC due to: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [42ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [43ms] sending RPC to server b123247401d540528f8324235e936260, [44ms] received response from server b123247401d540528f8324235e936260: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [45ms] delaying RPC due to: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [58ms] refreshing cache from master, [59ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [61ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [62ms] sending RPC to server b123247401d540528f8324235e936260, [64ms] received response from server b123247401d540528f8324235e936260: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [64ms] delaying RPC due to: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [138ms] refreshing cache from master, [139ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [141ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [143ms] sending RPC to server b123247401d540528f8324235e936260, [144ms] received response from server b123247401d540528f8324235e936260: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [144ms] delaying RPC due to: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [159ms] refreshing cache from master, [159ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [162ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=Deferred@472831192(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f87c036bed0@3e6d67cf -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@921948274 after retry RPC@397456024) -> (continuation of Deferred@1092348344 after retry RPC@1403893617) -> (continuation of Deferred@1724764315 after retry RPC@1483072780), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3dd39632 -> wakeup thread Time-limited test -> (continuation of Deferred@921948274 after retry RPC@397456024) -> (continuation of Deferred@1092348344 after retry RPC@1403893617) -> (continuation of Deferred@1724764315 after retry RPC@1483072780))) after lookup
02:03:42.189 [DEBUG - kudu-nio-1] (Connection.java:748) [peer b123247401d540528f8324235e936260(127.13.243.2:38235)] connecting to peer
02:03:42.189 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235
02:03:42.190 [DEBUG - kudu-nio-6] (Connection.java:709) [peer b123247401d540528f8324235e936260(127.13.243.2:38235)] cleaning up while in state CONNECTING due to: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235
02:03:42.190 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location b123247401d540528f8324235e936260(127.13.243.2:38235) for tablet 89c23a195e1e46c19d1cff4420b1f6e5: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235
02:03:42.987 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:03:42.987 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421),7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985),b123247401d540528f8324235e936260(127.13.243.2:38235)[L]]}]
02:03:42.988 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=89c23a195e1e46c19d1cff4420b1f6e5, attempt=11, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=962), Traces: [0ms] sending RPC to server b123247401d540528f8324235e936260, [0ms] delaying RPC due to: Illegal state: connection is terminated, [19ms] sending RPC to server b123247401d540528f8324235e936260, [21ms] received response from server b123247401d540528f8324235e936260: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [21ms] delaying RPC due to: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [42ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [43ms] sending RPC to server b123247401d540528f8324235e936260, [44ms] received response from server b123247401d540528f8324235e936260: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [45ms] delaying RPC due to: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [58ms] refreshing cache from master, [59ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [61ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [62ms] sending RPC to server b123247401d540528f8324235e936260, [64ms] received response from server b123247401d540528f8324235e936260: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [64ms] delaying RPC due to: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [138ms] refreshing cache from master, [139ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [141ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [143ms] sending RPC to server b123247401d540528f8324235e936260, [144ms] received response from server b123247401d540528f8324235e936260: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [144ms] delaying RPC due to: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [159ms] refreshing cache from master, [159ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [162ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [163ms] sending RPC to server b123247401d540528f8324235e936260, [165ms] received response from server b123247401d540528f8324235e936260: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [165ms] delaying RPC due to: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [959ms] refreshing cache from master, [959ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [962ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=Deferred@472831192(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f87c036bed0@3e6d67cf -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@921948274 after retry RPC@397456024) -> (continuation of Deferred@1092348344 after retry RPC@1403893617) -> (continuation of Deferred@1724764315 after retry RPC@1483072780) -> (continuation of Deferred@718745534 after retry RPC@1575875181), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3dd39632 -> wakeup thread Time-limited test -> (continuation of Deferred@921948274 after retry RPC@397456024) -> (continuation of Deferred@1092348344 after retry RPC@1403893617) -> (continuation of Deferred@1724764315 after retry RPC@1483072780) -> (continuation of Deferred@718745534 after retry RPC@1575875181))) after lookup
02:03:42.988 [DEBUG - kudu-nio-1] (Connection.java:748) [peer b123247401d540528f8324235e936260(127.13.243.2:38235)] connecting to peer
02:03:42.990 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235
02:03:42.990 [DEBUG - kudu-nio-7] (Connection.java:709) [peer b123247401d540528f8324235e936260(127.13.243.2:38235)] cleaning up while in state CONNECTING due to: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235
02:03:42.990 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location b123247401d540528f8324235e936260(127.13.243.2:38235) for tablet 89c23a195e1e46c19d1cff4420b1f6e5: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235
02:03:43.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:43.304685 15218 raft_consensus.cc:493] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b123247401d540528f8324235e936260)
02:03:43.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:43.304906 15218 raft_consensus.cc:515] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:03:43.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:43.305725 15218 leader_election.cc:290] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b123247401d540528f8324235e936260 (127.13.243.2:38235), 7f08c11f6dc94dc3aa69febd1051f2e9 (127.13.243.3:41985)
02:03:43.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:43.308139 14589 leader_election.cc:336] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b123247401d540528f8324235e936260 (127.13.243.2:38235): Network error: Client connection negotiation failed: client connection to 127.13.243.2:38235: connect: Connection refused (error 111)
02:03:43.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:43.313570 15154 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89c23a195e1e46c19d1cff4420b1f6e5" candidate_uuid: "6422ba2afbf742febb59b258c6ac5101" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" is_pre_election: true
02:03:43.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:43.313997 15154 raft_consensus.cc:2468] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6422ba2afbf742febb59b258c6ac5101 in term 1.
02:03:43.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:43.314954 14587 leader_election.cc:304] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6422ba2afbf742febb59b258c6ac5101, 7f08c11f6dc94dc3aa69febd1051f2e9; no voters: b123247401d540528f8324235e936260
02:03:43.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:43.315222 15218 raft_consensus.cc:2804] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [term 2 FOLLOWER]: Leader pre-election won for term 3
02:03:43.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:43.315342 15218 raft_consensus.cc:493] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [term 2 FOLLOWER]: Starting leader election (detected failure of leader b123247401d540528f8324235e936260)
02:03:43.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:43.315428 15218 raft_consensus.cc:3060] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [term 2 FOLLOWER]: Advancing to term 3
02:03:43.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:43.317404 15218 raft_consensus.cc:515] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:03:43.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:43.318056 15218 leader_election.cc:290] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [CANDIDATE]: Term 3 election: Requested vote from peers b123247401d540528f8324235e936260 (127.13.243.2:38235), 7f08c11f6dc94dc3aa69febd1051f2e9 (127.13.243.3:41985)
02:03:43.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:43.318663 15154 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89c23a195e1e46c19d1cff4420b1f6e5" candidate_uuid: "6422ba2afbf742febb59b258c6ac5101" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9"
02:03:43.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:43.318858 15154 raft_consensus.cc:3060] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [term 1 FOLLOWER]: Advancing to term 3
02:03:43.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:43.319566 14589 leader_election.cc:336] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer b123247401d540528f8324235e936260 (127.13.243.2:38235): Network error: Client connection negotiation failed: client connection to 127.13.243.2:38235: connect: Connection refused (error 111)
02:03:43.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:43.322656 15154 raft_consensus.cc:2468] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6422ba2afbf742febb59b258c6ac5101 in term 3.
02:03:43.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:43.323082 14587 leader_election.cc:304] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6422ba2afbf742febb59b258c6ac5101, 7f08c11f6dc94dc3aa69febd1051f2e9; no voters: b123247401d540528f8324235e936260
02:03:43.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:43.323441 15218 raft_consensus.cc:2804] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [term 3 FOLLOWER]: Leader election won for term 3
02:03:43.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:43.323877 15218 raft_consensus.cc:697] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [term 3 LEADER]: Becoming Leader. State: Replica: 6422ba2afbf742febb59b258c6ac5101, State: Running, Role: LEADER
02:03:43.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:43.324477 15218 consensus_queue.cc:237] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:03:43.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:43.327991 14467 catalog_manager.cc:5696] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 reported cstate change: term changed from 2 to 3, leader changed from b123247401d540528f8324235e936260 (127.13.243.2) to 6422ba2afbf742febb59b258c6ac5101 (127.13.243.1). New cstate: current_term: 3 leader_uuid: "6422ba2afbf742febb59b258c6ac5101" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } health_report { overall_health: UNKNOWN } } }
02:03:43.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:43.749054 15154 raft_consensus.cc:1275] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [term 3 FOLLOWER]: Refusing update from remote peer 6422ba2afbf742febb59b258c6ac5101: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
02:03:43.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:43.749984 15218 consensus_queue.cc:1048] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 4, Time since last communication: 0.000s
02:03:43.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:43.765714 14589 consensus_peers.cc:597] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 -> Peer b123247401d540528f8324235e936260 (127.13.243.2:38235): Couldn't send request to peer b123247401d540528f8324235e936260. Status: Network error: Client connection negotiation failed: client connection to 127.13.243.2:38235: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
02:03:45.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:45.989928 14589 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.243.2:38235: connect: Connection refused (error 111) [suppressed 7 similar messages]
02:03:45.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:45.991590 14589 consensus_peers.cc:597] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 -> Peer b123247401d540528f8324235e936260 (127.13.243.2:38235): Couldn't send request to peer b123247401d540528f8324235e936260. Status: Network error: Client connection negotiation failed: client connection to 127.13.243.2:38235: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
02:03:46.208 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:03:46.208 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)[L],7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985),b123247401d540528f8324235e936260(127.13.243.2:38235)]}]
02:03:46.208 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=89c23a195e1e46c19d1cff4420b1f6e5, attempt=13, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=4183), Traces: [0ms] sending RPC to server b123247401d540528f8324235e936260, [0ms] delaying RPC due to: Illegal state: connection is terminated, [19ms] sending RPC to server b123247401d540528f8324235e936260, [21ms] received response from server b123247401d540528f8324235e936260: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [21ms] delaying RPC due to: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [42ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [43ms] sending RPC to server b123247401d540528f8324235e936260, [44ms] received response from server b123247401d540528f8324235e936260: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [45ms] delaying RPC due to: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [58ms] refreshing cache from master, [59ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [61ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [62ms] sending RPC to server b123247401d540528f8324235e936260, [64ms] received response from server b123247401d540528f8324235e936260: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [64ms] delaying RPC due to: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [138ms] refreshing cache from master, [139ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [141ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [143ms] sending RPC to server b123247401d540528f8324235e936260, [144ms] received response from server b123247401d540528f8324235e936260: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [144ms] delaying RPC due to: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [159ms] refreshing cache from master, [159ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [162ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [163ms] sending RPC to server b123247401d540528f8324235e936260, [165ms] received response from server b123247401d540528f8324235e936260: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [165ms] delaying RPC due to: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [959ms] refreshing cache from master, [959ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [962ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [963ms] sending RPC to server b123247401d540528f8324235e936260, [965ms] received response from server b123247401d540528f8324235e936260: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [965ms] delaying RPC due to: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [4179ms] refreshing cache from master, [4180ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [4182ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=Deferred@472831192(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f87c036bed0@3e6d67cf -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@921948274 after retry RPC@397456024) -> (continuation of Deferred@1092348344 after retry RPC@1403893617) -> (continuation of Deferred@1724764315 after retry RPC@1483072780) -> (continuation of Deferred@718745534 after retry RPC@1575875181) -> (continuation of Deferred@718008181 after retry RPC@486582136), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3dd39632 -> wakeup thread Time-limited test -> (continuation of Deferred@921948274 after retry RPC@397456024) -> (continuation of Deferred@1092348344 after retry RPC@1403893617) -> (continuation of Deferred@1724764315 after retry RPC@1483072780) -> (continuation of Deferred@718745534 after retry RPC@1575875181) -> (continuation of Deferred@718008181 after retry RPC@486582136))) after lookup
02:03:46.209 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)] connecting to peer
02:03:46.210 [DEBUG - kudu-nio-0] (Connection.java:757) [peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)] Successfully connected to peer
02:03:46.232 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0x93d29398, L:/127.0.0.1:59710 - R:/127.13.243.1:33421] using TOKEN/null
02:03:46.258 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.13.243.2:38235
02:03:46.259 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

02:03:46.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:46.260393 14284 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu
02:03:46.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu
02:03:46.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-1/wal
02:03:46.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-1/data
02:03:46.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:03:46.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:03:46.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:03:46.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:03:46.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:03:46.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:03:46.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-1/logs
02:03:46.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-1/data/info.pb
02:03:46.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:03:46.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:03:46.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:03:46.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:03:46.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:03:46.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:03:46.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:03:46.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:03:46.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:03:46.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:03:46.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.13.243.2:38235
02:03:46.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.13.243.2
02:03:46.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.13.243.62:45883,127.13.243.61:44961,127.13.243.60:35015
02:03:46.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42383
02:03:46.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.13.243.2
02:03:46.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.13.243.20:34675
02:03:46.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:03:46.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:03:46.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:03:46.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:46.588528 15256 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:03:46.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:46.588876 15256 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:03:46.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:46.588973 15256 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:03:46.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:46.597016 15256 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:03:46.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:46.597220 15256 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.243.2
02:03:46.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:46.607940 15256 tablet_server_runner.cc:78] Tablet server non-default flags:
02:03:46.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:03:46.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.13.243.20:34675
02:03:46.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:03:46.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:03:46.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-1/data
02:03:46.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-1/wal
02:03:46.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:03:46.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:03:46.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.13.243.2:38235
02:03:46.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:03:46.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:03:46.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:03:46.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-1/data/info.pb
02:03:46.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.13.243.2
02:03:46.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42383
02:03:46.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.13.243.62:45883,127.13.243.61:44961,127.13.243.60:35015
02:03:46.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:03:46.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:03:46.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:03:46.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:03:46.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:03:46.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.13.243.2
02:03:46.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-1/logs
02:03:46.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:03:46.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:03:46.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:03:46.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:03:46.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:03:46.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision a3f35afc2b6dad6aa403ec58378221544a26edd0
02:03:46.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:03:46.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:40:41 UTC on bdcb31816ec0
02:03:46.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 16849
02:03:46.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
02:03:46.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:46.609916 15256 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:03:46.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:46.612011 15256 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:03:46.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:46.621995 15262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:46.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:46.622028 15261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:46.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:46.622476 15264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:46.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:46.623376 15256 server_base.cc:1061] running on GCE node
02:03:46.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:46.624033 15256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:03:46.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:46.624986 15256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:03:46.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:46.626219 15256 hybrid_clock.cc:648] HybridClock initialized: now 1783476226626168 us; error 42 us; skew 500 ppm
02:03:46.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:46.626583 15256 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:03:46.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:46.628831 15256 webserver.cc:534] Webserver started at http://127.13.243.2:42383/ using document root <none> and password file <none>
02:03:46.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:46.629578 15256 fs_manager.cc:362] Metadata directory not provided
02:03:46.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:46.629724 15256 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:03:46.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:46.635010 15256 fs_manager.cc:714] Time spent opening directory manager: real 0.003s	user 0.004s	sys 0.000s
02:03:46.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:46.637784 15270 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:03:46.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:46.639097 15256 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
02:03:46.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:46.639230 15256 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-1/data,/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-1/wal
02:03:46.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "b123247401d540528f8324235e936260"
02:03:46.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:03:37 on dist-test-slave-pjt5"
02:03:46.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:46.639863 15256 fs_report.cc:389] FS layout report
02:03:46.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:03:46.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-1/wal
02:03:46.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-1/wal
02:03:46.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-1/data/data
02:03:46.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:03:46.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:03:46.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:03:46.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:03:46.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:03:46.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:03:46.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:03:46.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:03:46.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:03:46.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:03:46.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:03:46.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:46.667811 15256 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:03:46.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:46.668644 15256 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:03:46.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:46.668881 15256 kserver.cc:163] Server-wide thread pool size limit: 3276
02:03:46.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:46.669790 15256 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:03:46.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:46.671839 15277 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
02:03:46.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:46.677639 15256 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
02:03:46.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:46.677719 15256 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.006s	user 0.000s	sys 0.001s
02:03:46.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:46.677814 15256 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
02:03:46.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:46.680691 15256 ts_tablet_manager.cc:616] Registered 1 tablets
02:03:46.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:46.680778 15256 ts_tablet_manager.cc:595] Time spent register tablets: real 0.003s	user 0.000s	sys 0.001s
02:03:46.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:46.681301 15277 tablet_bootstrap.cc:492] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260: Bootstrap starting.
02:03:46.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:46.704775 15277 log.cc:826] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260: Log is configured to *not* fsync() on all Append() calls
02:03:46.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:46.723459 15277 tablet_bootstrap.cc:492] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=6 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
02:03:46.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:46.724093 15277 tablet_bootstrap.cc:492] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260: Bootstrap complete.
02:03:46.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:46.725337 15277 ts_tablet_manager.cc:1403] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260: Time spent bootstrapping tablet: real 0.044s	user 0.033s	sys 0.008s
02:03:46.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:46.729701 15256 rpc_server.cc:307] RPC server started. Bound to: 127.13.243.2:38235
02:03:46.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:46.729822 15384 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.243.2:38235 every 8 connection(s)
02:03:46.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:46.731271 15256 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-1/data/info.pb
02:03:46.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:46.732209 15277 raft_consensus.cc:359] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:03:46.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:46.733202 15277 raft_consensus.cc:740] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b123247401d540528f8324235e936260, State: Initialized, Role: FOLLOWER
02:03:46.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:46.734046 15277 consensus_queue.cc:260] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:03:46.742 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:03:46.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:46.740693 14284 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu as pid 15256
02:03:46.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:46.758459 15277 ts_tablet_manager.cc:1434] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260: Time spent starting tablet: real 0.033s	user 0.008s	sys 0.010s
02:03:46.768 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 9 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291, tablet=89c23a195e1e46c19d1cff4420b1f6e5, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:03:46.770 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.13.243.1:33421
02:03:46.771 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

02:03:46.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:46.771384 14284 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu pid 14549
02:03:46.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:46.781858 15386 heartbeater.cc:344] Connected to a master server at 127.13.243.62:45883
02:03:46.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:46.782343 15386 heartbeater.cc:461] Registering TS with master...
02:03:46.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:46.782815 15387 heartbeater.cc:344] Connected to a master server at 127.13.243.61:44961
02:03:46.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:46.783116 15387 heartbeater.cc:461] Registering TS with master...
02:03:46.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:46.783558 15386 heartbeater.cc:507] Master 127.13.243.62:45883 requested a full tablet report, sending...
02:03:46.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:46.783991 15387 heartbeater.cc:507] Master 127.13.243.61:44961 requested a full tablet report, sending...
02:03:46.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:46.786769 14397 ts_manager.cc:194] Re-registered known tserver with Master: b123247401d540528f8324235e936260 (127.13.243.2:38235)
02:03:46.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:46.786948 14325 ts_manager.cc:194] Re-registered known tserver with Master: b123247401d540528f8324235e936260 (127.13.243.2:38235)
02:03:46.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:46.790402 15385 heartbeater.cc:344] Connected to a master server at 127.13.243.60:35015
02:03:46.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:46.790561 15385 heartbeater.cc:461] Registering TS with master...
02:03:46.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:46.791153 15385 heartbeater.cc:507] Master 127.13.243.60:35015 requested a full tablet report, sending...
02:03:46.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:46.792429 14467 ts_manager.cc:194] Re-registered known tserver with Master: b123247401d540528f8324235e936260 (127.13.243.2:38235)
02:03:46.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:46.793953 14467 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.13.243.2:34069
02:03:46.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:46.796545 15385 heartbeater.cc:499] Master 127.13.243.60:35015 was elected leader, sending a full tablet report...
02:03:46.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:46.800737 14694 generic_service.cc:196] Checking for leaks (request via RPC)
02:03:46.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:46.899742 14284 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu with pid 14549
02:03:46.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:46.919116 15086 connection.cc:573] server connection from 127.13.243.1:37951 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
02:03:46.919 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:03:46.920 [DEBUG - kudu-nio-0] (Connection.java:239) [peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)] handling channelInactive
02:03:46.920 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)] cleaning up while in state READY due to: connection closed
02:03:46.924 [DEBUG - pool-2-thread-1] (Connection.java:748) [peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)] connecting to peer
02:03:46.925 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421
02:03:46.925 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)] cleaning up while in state CONNECTING due to: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421
02:03:46.926 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421) for tablet 89c23a195e1e46c19d1cff4420b1f6e5: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421
02:03:46.947 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:03:46.948 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)[L],7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985),b123247401d540528f8324235e936260(127.13.243.2:38235)]}]
02:03:46.948 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=89c23a195e1e46c19d1cff4420b1f6e5, attempt=3, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=28), Traces: [0ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [0ms] delaying RPC due to: Illegal state: connection is terminated, [4ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [6ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [6ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [27ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=Deferred@646931394(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f87c036bed0@7ea0da61 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7ce112bf -> wakeup thread Time-limited test)) after lookup
02:03:46.948 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)] connecting to peer
02:03:46.949 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421
02:03:46.949 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)] cleaning up while in state CONNECTING due to: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421
02:03:46.949 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421) for tablet 89c23a195e1e46c19d1cff4420b1f6e5: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421
02:03:46.987 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:03:46.987 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)[L],7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985),b123247401d540528f8324235e936260(127.13.243.2:38235)]}]
02:03:46.988 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=89c23a195e1e46c19d1cff4420b1f6e5, attempt=5, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=68), Traces: [0ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [0ms] delaying RPC due to: Illegal state: connection is terminated, [4ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [6ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [6ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [27ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [28ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [29ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [29ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [67ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=Deferred@646931394(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f87c036bed0@7ea0da61 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@920842945 after retry RPC@394535276), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7ce112bf -> wakeup thread Time-limited test -> (continuation of Deferred@920842945 after retry RPC@394535276))) after lookup
02:03:46.988 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)] connecting to peer
02:03:46.989 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421
02:03:46.989 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)] cleaning up while in state CONNECTING due to: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421
02:03:46.989 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421) for tablet 89c23a195e1e46c19d1cff4420b1f6e5: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421
02:03:47.027 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:03:47.028 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)[L],7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985),b123247401d540528f8324235e936260(127.13.243.2:38235)]}]
02:03:47.028 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=89c23a195e1e46c19d1cff4420b1f6e5, attempt=7, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=108), Traces: [0ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [0ms] delaying RPC due to: Illegal state: connection is terminated, [4ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [6ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [6ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [27ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [28ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [29ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [29ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [67ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [68ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [69ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [70ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [104ms] refreshing cache from master, [104ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [107ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=Deferred@646931394(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f87c036bed0@7ea0da61 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@920842945 after retry RPC@394535276) -> (continuation of Deferred@1175072227 after retry RPC@1882450703), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7ce112bf -> wakeup thread Time-limited test -> (continuation of Deferred@920842945 after retry RPC@394535276) -> (continuation of Deferred@1175072227 after retry RPC@1882450703))) after lookup
02:03:47.029 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)] connecting to peer
02:03:47.030 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421
02:03:47.030 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)] cleaning up while in state CONNECTING due to: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421
02:03:47.031 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421) for tablet 89c23a195e1e46c19d1cff4420b1f6e5: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421
02:03:47.128 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:03:47.128 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)[L],7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985),b123247401d540528f8324235e936260(127.13.243.2:38235)]}]
02:03:47.128 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=89c23a195e1e46c19d1cff4420b1f6e5, attempt=9, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=208), Traces: [0ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [0ms] delaying RPC due to: Illegal state: connection is terminated, [4ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [6ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [6ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [27ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [28ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [29ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [29ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [67ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [68ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [69ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [70ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [104ms] refreshing cache from master, [104ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [107ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [109ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [110ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [111ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [204ms] refreshing cache from master, [204ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [208ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=Deferred@646931394(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f87c036bed0@7ea0da61 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@920842945 after retry RPC@394535276) -> (continuation of Deferred@1175072227 after retry RPC@1882450703) -> (continuation of Deferred@1777288617 after retry RPC@116109320), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7ce112bf -> wakeup thread Time-limited test -> (continuation of Deferred@920842945 after retry RPC@394535276) -> (continuation of Deferred@1175072227 after retry RPC@1882450703) -> (continuation of Deferred@1777288617 after retry RPC@116109320))) after lookup
02:03:47.129 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)] connecting to peer
02:03:47.130 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421
02:03:47.130 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)] cleaning up while in state CONNECTING due to: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421
02:03:47.130 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421) for tablet 89c23a195e1e46c19d1cff4420b1f6e5: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421
02:03:48.107 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:03:48.107 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)[L],7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985),b123247401d540528f8324235e936260(127.13.243.2:38235)]}]
02:03:48.108 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=89c23a195e1e46c19d1cff4420b1f6e5, attempt=11, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=1188), Traces: [0ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [0ms] delaying RPC due to: Illegal state: connection is terminated, [4ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [6ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [6ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [27ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [28ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [29ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [29ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [67ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [68ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [69ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [70ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [104ms] refreshing cache from master, [104ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [107ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [109ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [110ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [111ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [204ms] refreshing cache from master, [204ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [208ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [209ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [210ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [210ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [1183ms] refreshing cache from master, [1184ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [1187ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=Deferred@646931394(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f87c036bed0@7ea0da61 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@920842945 after retry RPC@394535276) -> (continuation of Deferred@1175072227 after retry RPC@1882450703) -> (continuation of Deferred@1777288617 after retry RPC@116109320) -> (continuation of Deferred@1804612791 after retry RPC@815837757), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7ce112bf -> wakeup thread Time-limited test -> (continuation of Deferred@920842945 after retry RPC@394535276) -> (continuation of Deferred@1175072227 after retry RPC@1882450703) -> (continuation of Deferred@1777288617 after retry RPC@116109320) -> (continuation of Deferred@1804612791 after retry RPC@815837757))) after lookup
02:03:48.108 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)] connecting to peer
02:03:48.109 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421
02:03:48.109 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)] cleaning up while in state CONNECTING due to: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421
02:03:48.110 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421) for tablet 89c23a195e1e46c19d1cff4420b1f6e5: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421
02:03:48.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:48.196738 15400 raft_consensus.cc:493] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
02:03:48.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:48.197024 15400 raft_consensus.cc:515] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:03:48.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:48.198626 15400 leader_election.cc:290] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6422ba2afbf742febb59b258c6ac5101 (127.13.243.1:33421), 7f08c11f6dc94dc3aa69febd1051f2e9 (127.13.243.3:41985)
02:03:48.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:48.200296 15271 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.243.1:33421: connect: Connection refused (error 111)
02:03:48.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:48.202183 15271 leader_election.cc:336] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6422ba2afbf742febb59b258c6ac5101 (127.13.243.1:33421): Network error: Client connection negotiation failed: client connection to 127.13.243.1:33421: connect: Connection refused (error 111)
02:03:48.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:48.206089 15154 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89c23a195e1e46c19d1cff4420b1f6e5" candidate_uuid: "b123247401d540528f8324235e936260" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" is_pre_election: true
02:03:48.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:48.206996 15271 leader_election.cc:304] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b123247401d540528f8324235e936260; no voters: 6422ba2afbf742febb59b258c6ac5101, 7f08c11f6dc94dc3aa69febd1051f2e9
02:03:48.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:48.207418 15400 raft_consensus.cc:2749] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
02:03:48.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:48.400794 15405 raft_consensus.cc:493] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6422ba2afbf742febb59b258c6ac5101)
02:03:48.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:48.401008 15405 raft_consensus.cc:515] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:03:48.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:48.402698 15405 leader_election.cc:290] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6422ba2afbf742febb59b258c6ac5101 (127.13.243.1:33421), b123247401d540528f8324235e936260 (127.13.243.2:38235)
02:03:48.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:48.404222 15085 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.243.1:33421: connect: Connection refused (error 111)
02:03:48.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:48.405900 15085 leader_election.cc:336] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6422ba2afbf742febb59b258c6ac5101 (127.13.243.1:33421): Network error: Client connection negotiation failed: client connection to 127.13.243.1:33421: connect: Connection refused (error 111)
02:03:48.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:48.411214 15340 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89c23a195e1e46c19d1cff4420b1f6e5" candidate_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "b123247401d540528f8324235e936260" is_pre_election: true
02:03:48.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:48.411572 15340 raft_consensus.cc:2468] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f08c11f6dc94dc3aa69febd1051f2e9 in term 2.
02:03:48.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:48.412504 15087 leader_election.cc:304] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7f08c11f6dc94dc3aa69febd1051f2e9, b123247401d540528f8324235e936260; no voters: 6422ba2afbf742febb59b258c6ac5101
02:03:48.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:48.412883 15405 raft_consensus.cc:2804] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [term 3 FOLLOWER]: Leader pre-election won for term 4
02:03:48.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:48.413020 15405 raft_consensus.cc:493] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6422ba2afbf742febb59b258c6ac5101)
02:03:48.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:48.413156 15405 raft_consensus.cc:3060] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [term 3 FOLLOWER]: Advancing to term 4
02:03:48.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:48.415233 15405 raft_consensus.cc:515] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:03:48.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:48.415845 15405 leader_election.cc:290] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [CANDIDATE]: Term 4 election: Requested vote from peers 6422ba2afbf742febb59b258c6ac5101 (127.13.243.1:33421), b123247401d540528f8324235e936260 (127.13.243.2:38235)
02:03:48.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:48.416500 15340 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89c23a195e1e46c19d1cff4420b1f6e5" candidate_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "b123247401d540528f8324235e936260"
02:03:48.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:48.416738 15340 raft_consensus.cc:3060] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [term 2 FOLLOWER]: Advancing to term 4
02:03:48.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:48.417202 15085 leader_election.cc:336] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 6422ba2afbf742febb59b258c6ac5101 (127.13.243.1:33421): Network error: Client connection negotiation failed: client connection to 127.13.243.1:33421: connect: Connection refused (error 111)
02:03:48.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:48.420634 15340 raft_consensus.cc:2468] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f08c11f6dc94dc3aa69febd1051f2e9 in term 4.
02:03:48.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:48.421149 15087 leader_election.cc:304] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7f08c11f6dc94dc3aa69febd1051f2e9, b123247401d540528f8324235e936260; no voters: 6422ba2afbf742febb59b258c6ac5101
02:03:48.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:48.421461 15405 raft_consensus.cc:2804] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [term 4 FOLLOWER]: Leader election won for term 4
02:03:48.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:48.421849 15405 raft_consensus.cc:697] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [term 4 LEADER]: Becoming Leader. State: Replica: 7f08c11f6dc94dc3aa69febd1051f2e9, State: Running, Role: LEADER
02:03:48.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:48.422499 15405 consensus_queue.cc:237] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 12, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:03:48.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:48.426117 14467 catalog_manager.cc:5696] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 reported cstate change: term changed from 3 to 4, leader changed from 6422ba2afbf742febb59b258c6ac5101 (127.13.243.1) to 7f08c11f6dc94dc3aa69febd1051f2e9 (127.13.243.3). New cstate: current_term: 4 leader_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } health_report { overall_health: HEALTHY } } }
02:03:48.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:48.896472 15340 raft_consensus.cc:1275] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [term 4 FOLLOWER]: Refusing update from remote peer 7f08c11f6dc94dc3aa69febd1051f2e9: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 4 index: 13. (index mismatch)
02:03:48.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:48.897526 15405 consensus_queue.cc:1048] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 8, Time since last communication: 0.000s
02:03:48.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:48.912686 15085 consensus_peers.cc:597] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 -> Peer 6422ba2afbf742febb59b258c6ac5101 (127.13.243.1:33421): Couldn't send request to peer 6422ba2afbf742febb59b258c6ac5101. Status: Network error: Client connection negotiation failed: client connection to 127.13.243.1:33421: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
02:03:51.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:51.504283 15085 consensus_peers.cc:597] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 -> Peer 6422ba2afbf742febb59b258c6ac5101 (127.13.243.1:33421): Couldn't send request to peer 6422ba2afbf742febb59b258c6ac5101. Status: Network error: Client connection negotiation failed: client connection to 127.13.243.1:33421: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
02:03:51.648 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:03:51.648 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421),7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)[L],b123247401d540528f8324235e936260(127.13.243.2:38235)]}]
02:03:51.648 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=89c23a195e1e46c19d1cff4420b1f6e5, attempt=13, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=4728), Traces: [0ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [0ms] delaying RPC due to: Illegal state: connection is terminated, [4ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [6ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [6ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [27ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [28ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [29ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [29ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [67ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [68ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [69ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [70ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [104ms] refreshing cache from master, [104ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [107ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [109ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [110ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [111ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [204ms] refreshing cache from master, [204ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [208ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [209ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [210ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [210ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [1183ms] refreshing cache from master, [1184ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [1187ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [1188ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [1189ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [1190ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [4724ms] refreshing cache from master, [4725ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [4727ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=Deferred@646931394(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f87c036bed0@7ea0da61 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@920842945 after retry RPC@394535276) -> (continuation of Deferred@1175072227 after retry RPC@1882450703) -> (continuation of Deferred@1777288617 after retry RPC@116109320) -> (continuation of Deferred@1804612791 after retry RPC@815837757) -> (continuation of Deferred@1035875058 after retry RPC@138038818), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7ce112bf -> wakeup thread Time-limited test -> (continuation of Deferred@920842945 after retry RPC@394535276) -> (continuation of Deferred@1175072227 after retry RPC@1882450703) -> (continuation of Deferred@1777288617 after retry RPC@116109320) -> (continuation of Deferred@1804612791 after retry RPC@815837757) -> (continuation of Deferred@1035875058 after retry RPC@138038818))) after lookup
02:03:51.649 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)] connecting to peer
02:03:51.650 [DEBUG - kudu-nio-7] (Connection.java:757) [peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)] Successfully connected to peer
02:03:51.683 [DEBUG - kudu-nio-7] (Negotiator.java:919) Authenticated connection [id: 0xe7434b88, L:/127.0.0.1:37258 - R:/127.13.243.3:41985] using TOKEN/null
02:03:51.708 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.13.243.1:33421
02:03:51.708 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

02:03:51.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:51.709395 14284 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu
02:03:51.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu
02:03:51.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-0/wal
02:03:51.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-0/data
02:03:51.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:03:51.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:03:51.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:03:51.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:03:51.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:03:51.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:03:51.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-0/logs
02:03:51.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-0/data/info.pb
02:03:51.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:03:51.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:03:51.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:03:51.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:03:51.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:03:51.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:03:51.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:03:51.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:03:51.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:03:51.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:03:51.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.13.243.1:33421
02:03:51.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.13.243.1
02:03:51.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.13.243.62:45883,127.13.243.61:44961,127.13.243.60:35015
02:03:51.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42315
02:03:51.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.13.243.1
02:03:51.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.13.243.20:34675
02:03:51.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:03:51.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:03:51.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:03:52.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:52.046854 15449 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:03:52.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:52.047128 15449 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:03:52.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:52.047226 15449 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:03:52.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:52.055398 15449 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:03:52.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:52.055557 15449 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.243.1
02:03:52.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:52.065606 15449 tablet_server_runner.cc:78] Tablet server non-default flags:
02:03:52.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:03:52.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.13.243.20:34675
02:03:52.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:03:52.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:03:52.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-0/data
02:03:52.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-0/wal
02:03:52.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:03:52.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:03:52.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.13.243.1:33421
02:03:52.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:03:52.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:03:52.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:03:52.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-0/data/info.pb
02:03:52.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.13.243.1
02:03:52.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42315
02:03:52.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.13.243.62:45883,127.13.243.61:44961,127.13.243.60:35015
02:03:52.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:03:52.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:03:52.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:03:52.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:03:52.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:03:52.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.13.243.1
02:03:52.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-0/logs
02:03:52.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:03:52.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:03:52.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:03:52.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:03:52.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:03:52.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision a3f35afc2b6dad6aa403ec58378221544a26edd0
02:03:52.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:03:52.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:40:41 UTC on bdcb31816ec0
02:03:52.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 16849
02:03:52.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
02:03:52.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:52.067376 15449 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:03:52.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:52.069133 15449 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:03:52.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:52.078853 15455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:52.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:52.078886 15454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:52.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:52.079737 15457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:52.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:52.080451 15449 server_base.cc:1061] running on GCE node
02:03:52.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:52.081096 15449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:03:52.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:52.082057 15449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:03:52.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:52.083285 15449 hybrid_clock.cc:648] HybridClock initialized: now 1783476232083226 us; error 50 us; skew 500 ppm
02:03:52.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:52.083658 15449 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:03:52.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:52.085989 15449 webserver.cc:534] Webserver started at http://127.13.243.1:42315/ using document root <none> and password file <none>
02:03:52.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:52.086884 15449 fs_manager.cc:362] Metadata directory not provided
02:03:52.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:52.087035 15449 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:03:52.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:52.092731 15449 fs_manager.cc:714] Time spent opening directory manager: real 0.004s	user 0.005s	sys 0.000s
02:03:52.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:52.095731 15463 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:03:52.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:52.097007 15449 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
02:03:52.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:52.097223 15449 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-0/data,/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-0/wal
02:03:52.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "6422ba2afbf742febb59b258c6ac5101"
02:03:52.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:03:36 on dist-test-slave-pjt5"
02:03:52.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:52.097931 15449 fs_report.cc:389] FS layout report
02:03:52.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:03:52.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-0/wal
02:03:52.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-0/wal
02:03:52.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-0/data/data
02:03:52.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:03:52.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:03:52.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:03:52.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:03:52.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:03:52.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:03:52.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:03:52.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:03:52.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:03:52.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:03:52.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:03:52.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:52.113152 15449 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:03:52.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:52.114064 15449 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:03:52.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:52.114440 15449 kserver.cc:163] Server-wide thread pool size limit: 3276
02:03:52.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:52.115337 15449 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:03:52.117 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:52.117262 15472 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
02:03:52.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:52.123138 15449 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
02:03:52.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:52.123215 15449 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.006s	user 0.001s	sys 0.000s
02:03:52.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:52.123310 15449 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
02:03:52.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:52.126209 15449 ts_tablet_manager.cc:616] Registered 1 tablets
02:03:52.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:52.126421 15449 ts_tablet_manager.cc:595] Time spent register tablets: real 0.003s	user 0.002s	sys 0.001s
02:03:52.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:52.126785 15472 tablet_bootstrap.cc:492] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101: Bootstrap starting.
02:03:52.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:52.150594 15472 log.cc:826] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101: Log is configured to *not* fsync() on all Append() calls
02:03:52.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:52.170842 15449 rpc_server.cc:307] RPC server started. Bound to: 127.13.243.1:33421
02:03:52.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:52.170989 15579 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.243.1:33421 every 8 connection(s)
02:03:52.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:52.172392 15449 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-0/data/info.pb
02:03:52.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:52.173708 15472 tablet_bootstrap.cc:492] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=9 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
02:03:52.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:52.174512 15472 tablet_bootstrap.cc:492] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101: Bootstrap complete.
02:03:52.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:52.177392 14284 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu as pid 15449
02:03:52.177 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:03:52.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:52.185106 15472 ts_tablet_manager.cc:1403] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101: Time spent bootstrapping tablet: real 0.059s	user 0.033s	sys 0.016s
02:03:52.206 [DEBUG - kudu-nio-7] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 12 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291, tablet=89c23a195e1e46c19d1cff4420b1f6e5, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:03:52.214 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.13.243.3:41985
02:03:52.214 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

02:03:52.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:52.215186 14284 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu pid 15068
02:03:52.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:52.201182 15472 raft_consensus.cc:359] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:03:52.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:52.229656 15472 raft_consensus.cc:740] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6422ba2afbf742febb59b258c6ac5101, State: Initialized, Role: FOLLOWER
02:03:52.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:52.230541 15472 consensus_queue.cc:260] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 3.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:03:52.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:52.230857 15582 heartbeater.cc:344] Connected to a master server at 127.13.243.61:44961
02:03:52.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:52.231257 15582 heartbeater.cc:461] Registering TS with master...
02:03:52.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:52.231454 15581 heartbeater.cc:344] Connected to a master server at 127.13.243.62:45883
02:03:52.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:52.231619 15581 heartbeater.cc:461] Registering TS with master...
02:03:52.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:52.232511 15582 heartbeater.cc:507] Master 127.13.243.61:44961 requested a full tablet report, sending...
02:03:52.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:52.235428 15581 heartbeater.cc:507] Master 127.13.243.62:45883 requested a full tablet report, sending...
02:03:52.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:52.236830 15580 heartbeater.cc:344] Connected to a master server at 127.13.243.60:35015
02:03:52.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:52.236994 15580 heartbeater.cc:461] Registering TS with master...
02:03:52.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:52.237553 15580 heartbeater.cc:507] Master 127.13.243.60:35015 requested a full tablet report, sending...
02:03:52.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:52.239223 14326 ts_manager.cc:194] Re-registered known tserver with Master: 6422ba2afbf742febb59b258c6ac5101 (127.13.243.1:33421)
02:03:52.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:52.239771 15472 ts_tablet_manager.cc:1434] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101: Time spent starting tablet: real 0.055s	user 0.002s	sys 0.017s
02:03:52.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:52.240090 14396 ts_manager.cc:194] Re-registered known tserver with Master: 6422ba2afbf742febb59b258c6ac5101 (127.13.243.1:33421)
02:03:52.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:52.242694 14467 ts_manager.cc:194] Re-registered known tserver with Master: 6422ba2afbf742febb59b258c6ac5101 (127.13.243.1:33421)
02:03:52.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:52.244233 14467 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.13.243.1:33601
02:03:52.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:52.246896 15580 heartbeater.cc:499] Master 127.13.243.60:35015 was elected leader, sending a full tablet report...
02:03:52.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:52.248457 15194 generic_service.cc:196] Checking for leaks (request via RPC)
02:03:52.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:52.349239 14284 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu with pid 15068
02:03:52.367 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:03:52.368 [DEBUG - kudu-nio-7] (Connection.java:239) [peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)] handling channelInactive
02:03:52.368 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)] cleaning up while in state READY due to: connection closed
02:03:52.369 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985) for tablet 89c23a195e1e46c19d1cff4420b1f6e5: connection closed
02:03:52.387 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:03:52.387 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421),7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)[L],b123247401d540528f8324235e936260(127.13.243.2:38235)]}]
02:03:52.387 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=89c23a195e1e46c19d1cff4420b1f6e5, attempt=2, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=19), Traces: [0ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [0ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: connection closed, [1ms] delaying RPC due to: Network error: connection closed, [16ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [19ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=Deferred@21336379(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f87c036bed0@5bd4846f -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@57cb54dc -> wakeup thread Time-limited test)) after lookup
02:03:52.388 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)] connecting to peer
02:03:52.389 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:03:52.389 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:03:52.389 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985) for tablet 89c23a195e1e46c19d1cff4420b1f6e5: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:03:52.406 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:03:52.407 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421),7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)[L],b123247401d540528f8324235e936260(127.13.243.2:38235)]}]
02:03:52.407 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=89c23a195e1e46c19d1cff4420b1f6e5, attempt=4, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=38), Traces: [0ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [0ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: connection closed, [1ms] delaying RPC due to: Network error: connection closed, [16ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [19ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [20ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [21ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [21ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [35ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [38ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=Deferred@21336379(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f87c036bed0@5bd4846f -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1213272284 after retry RPC@1169281214), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@57cb54dc -> wakeup thread Time-limited test -> (continuation of Deferred@1213272284 after retry RPC@1169281214))) after lookup
02:03:52.407 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)] connecting to peer
02:03:52.408 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:03:52.408 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:03:52.408 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985) for tablet 89c23a195e1e46c19d1cff4420b1f6e5: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:03:52.447 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:03:52.447 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421),7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)[L],b123247401d540528f8324235e936260(127.13.243.2:38235)]}]
02:03:52.448 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=89c23a195e1e46c19d1cff4420b1f6e5, attempt=6, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=79), Traces: [0ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [0ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: connection closed, [1ms] delaying RPC due to: Network error: connection closed, [16ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [19ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [20ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [21ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [21ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [35ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [38ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [39ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [40ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [40ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [76ms] refreshing cache from master, [76ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [79ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=Deferred@21336379(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f87c036bed0@5bd4846f -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1213272284 after retry RPC@1169281214) -> (continuation of Deferred@698195794 after retry RPC@1217873062), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@57cb54dc -> wakeup thread Time-limited test -> (continuation of Deferred@1213272284 after retry RPC@1169281214) -> (continuation of Deferred@698195794 after retry RPC@1217873062))) after lookup
02:03:52.448 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)] connecting to peer
02:03:52.449 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:03:52.450 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:03:52.450 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985) for tablet 89c23a195e1e46c19d1cff4420b1f6e5: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:03:52.567 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:03:52.567 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421),7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)[L],b123247401d540528f8324235e936260(127.13.243.2:38235)]}]
02:03:52.568 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=89c23a195e1e46c19d1cff4420b1f6e5, attempt=8, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=199), Traces: [0ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [0ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: connection closed, [1ms] delaying RPC due to: Network error: connection closed, [16ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [19ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [20ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [21ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [21ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [35ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [38ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [39ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [40ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [40ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [76ms] refreshing cache from master, [76ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [79ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [80ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [82ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [82ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [195ms] refreshing cache from master, [196ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [199ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=Deferred@21336379(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f87c036bed0@5bd4846f -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1213272284 after retry RPC@1169281214) -> (continuation of Deferred@698195794 after retry RPC@1217873062) -> (continuation of Deferred@1056567075 after retry RPC@1278819886), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@57cb54dc -> wakeup thread Time-limited test -> (continuation of Deferred@1213272284 after retry RPC@1169281214) -> (continuation of Deferred@698195794 after retry RPC@1217873062) -> (continuation of Deferred@1056567075 after retry RPC@1278819886))) after lookup
02:03:52.568 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)] connecting to peer
02:03:52.570 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:03:52.570 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:03:52.570 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985) for tablet 89c23a195e1e46c19d1cff4420b1f6e5: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:03:52.667 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:03:52.667 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421),7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)[L],b123247401d540528f8324235e936260(127.13.243.2:38235)]}]
02:03:52.668 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=89c23a195e1e46c19d1cff4420b1f6e5, attempt=10, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=299), Traces: [0ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [0ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: connection closed, [1ms] delaying RPC due to: Network error: connection closed, [16ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [19ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [20ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [21ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [21ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [35ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [38ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [39ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [40ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [40ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [76ms] refreshing cache from master, [76ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [79ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [80ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [82ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [82ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [195ms] refreshing cache from master, [196ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [199ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [200ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [202ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [202ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [296ms] refreshing cache from master, [296ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [299ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=Deferred@21336379(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f87c036bed0@5bd4846f -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1213272284 after retry RPC@1169281214) -> (continuation of Deferred@698195794 after retry RPC@1217873062) -> (continuation of Deferred@1056567075 after retry RPC@1278819886) -> (continuation of Deferred@1022259665 after retry RPC@958652460), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@57cb54dc -> wakeup thread Time-limited test -> (continuation of Deferred@1213272284 after retry RPC@1169281214) -> (continuation of Deferred@698195794 after retry RPC@1217873062) -> (continuation of Deferred@1056567075 after retry RPC@1278819886) -> (continuation of Deferred@1022259665 after retry RPC@958652460))) after lookup
02:03:52.668 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)] connecting to peer
02:03:52.669 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:03:52.670 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:03:52.670 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985) for tablet 89c23a195e1e46c19d1cff4420b1f6e5: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:03:53.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:53.453121 15595 raft_consensus.cc:493] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
02:03:53.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:53.453459 15595 raft_consensus.cc:515] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:03:53.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:53.455243 15595 leader_election.cc:290] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b123247401d540528f8324235e936260 (127.13.243.2:38235), 7f08c11f6dc94dc3aa69febd1051f2e9 (127.13.243.3:41985)
02:03:53.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:53.457041 15464 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.243.3:41985: connect: Connection refused (error 111)
02:03:53.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:53.458981 15464 leader_election.cc:336] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 7f08c11f6dc94dc3aa69febd1051f2e9 (127.13.243.3:41985): Network error: Client connection negotiation failed: client connection to 127.13.243.3:41985: connect: Connection refused (error 111)
02:03:53.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:53.462252 15340 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89c23a195e1e46c19d1cff4420b1f6e5" candidate_uuid: "6422ba2afbf742febb59b258c6ac5101" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "b123247401d540528f8324235e936260" is_pre_election: true
02:03:53.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:53.463111 15466 leader_election.cc:304] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6422ba2afbf742febb59b258c6ac5101; no voters: 7f08c11f6dc94dc3aa69febd1051f2e9, b123247401d540528f8324235e936260
02:03:53.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:53.463472 15595 raft_consensus.cc:2749] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
02:03:53.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:53.698607 15600 raft_consensus.cc:493] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7f08c11f6dc94dc3aa69febd1051f2e9)
02:03:53.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:53.698799 15600 raft_consensus.cc:515] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:03:53.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:53.699527 15600 leader_election.cc:290] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6422ba2afbf742febb59b258c6ac5101 (127.13.243.1:33421), 7f08c11f6dc94dc3aa69febd1051f2e9 (127.13.243.3:41985)
02:03:53.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:53.701447 15271 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.243.3:41985: connect: Connection refused (error 111)
02:03:53.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:53.703275 15271 leader_election.cc:336] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 7f08c11f6dc94dc3aa69febd1051f2e9 (127.13.243.3:41985): Network error: Client connection negotiation failed: client connection to 127.13.243.3:41985: connect: Connection refused (error 111)
02:03:53.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:53.707595 15535 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89c23a195e1e46c19d1cff4420b1f6e5" candidate_uuid: "b123247401d540528f8324235e936260" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "6422ba2afbf742febb59b258c6ac5101" is_pre_election: true
02:03:53.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:53.707969 15535 raft_consensus.cc:2468] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b123247401d540528f8324235e936260 in term 3.
02:03:53.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:53.708773 15271 leader_election.cc:304] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6422ba2afbf742febb59b258c6ac5101, b123247401d540528f8324235e936260; no voters: 7f08c11f6dc94dc3aa69febd1051f2e9
02:03:53.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:53.709079 15600 raft_consensus.cc:2804] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [term 4 FOLLOWER]: Leader pre-election won for term 5
02:03:53.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:53.709220 15600 raft_consensus.cc:493] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7f08c11f6dc94dc3aa69febd1051f2e9)
02:03:53.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:53.709291 15600 raft_consensus.cc:3060] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [term 4 FOLLOWER]: Advancing to term 5
02:03:53.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:53.711349 15600 raft_consensus.cc:515] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:03:53.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:53.711941 15600 leader_election.cc:290] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [CANDIDATE]: Term 5 election: Requested vote from peers 6422ba2afbf742febb59b258c6ac5101 (127.13.243.1:33421), 7f08c11f6dc94dc3aa69febd1051f2e9 (127.13.243.3:41985)
02:03:53.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:53.712379 15535 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89c23a195e1e46c19d1cff4420b1f6e5" candidate_uuid: "b123247401d540528f8324235e936260" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "6422ba2afbf742febb59b258c6ac5101"
02:03:53.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:53.712571 15535 raft_consensus.cc:3060] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [term 3 FOLLOWER]: Advancing to term 5
02:03:53.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:53.713411 15271 leader_election.cc:336] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 7f08c11f6dc94dc3aa69febd1051f2e9 (127.13.243.3:41985): Network error: Client connection negotiation failed: client connection to 127.13.243.3:41985: connect: Connection refused (error 111)
02:03:53.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:53.716555 15535 raft_consensus.cc:2468] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b123247401d540528f8324235e936260 in term 5.
02:03:53.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:53.717015 15271 leader_election.cc:304] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6422ba2afbf742febb59b258c6ac5101, b123247401d540528f8324235e936260; no voters: 7f08c11f6dc94dc3aa69febd1051f2e9
02:03:53.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:53.717315 15600 raft_consensus.cc:2804] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [term 5 FOLLOWER]: Leader election won for term 5
02:03:53.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:53.717769 15600 raft_consensus.cc:697] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [term 5 LEADER]: Becoming Leader. State: Replica: b123247401d540528f8324235e936260, State: Running, Role: LEADER
02:03:53.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:53.718302 15600 consensus_queue.cc:237] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 4.16, Last appended by leader: 16, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:03:53.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:53.722142 14467 catalog_manager.cc:5696] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 reported cstate change: term changed from 4 to 5, leader changed from 7f08c11f6dc94dc3aa69febd1051f2e9 (127.13.243.3) to b123247401d540528f8324235e936260 (127.13.243.2). New cstate: current_term: 5 leader_uuid: "b123247401d540528f8324235e936260" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } health_report { overall_health: UNKNOWN } } }
02:03:54.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:54.170179 15535 raft_consensus.cc:1275] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [term 5 FOLLOWER]: Refusing update from remote peer b123247401d540528f8324235e936260: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 5 index: 17. (index mismatch)
02:03:54.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:54.171149 15600 consensus_queue.cc:1048] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 12, Time since last communication: 0.000s
02:03:54.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:54.188961 15271 consensus_peers.cc:597] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 -> Peer 7f08c11f6dc94dc3aa69febd1051f2e9 (127.13.243.3:41985): Couldn't send request to peer 7f08c11f6dc94dc3aa69febd1051f2e9. Status: Network error: Client connection negotiation failed: client connection to 127.13.243.3:41985: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
02:03:54.287 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:03:54.287 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421),7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985),b123247401d540528f8324235e936260(127.13.243.2:38235)[L]]}]
02:03:54.288 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=89c23a195e1e46c19d1cff4420b1f6e5, attempt=12, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=1919), Traces: [0ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [0ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: connection closed, [1ms] delaying RPC due to: Network error: connection closed, [16ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [19ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [20ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [21ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [21ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [35ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [38ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [39ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [40ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [40ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [76ms] refreshing cache from master, [76ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [79ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [80ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [82ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [82ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [195ms] refreshing cache from master, [196ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [199ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [200ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [202ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [202ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [296ms] refreshing cache from master, [296ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [299ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [300ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [302ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [302ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [1916ms] refreshing cache from master, [1916ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [1919ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=Deferred@21336379(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f87c036bed0@5bd4846f -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1213272284 after retry RPC@1169281214) -> (continuation of Deferred@698195794 after retry RPC@1217873062) -> (continuation of Deferred@1056567075 after retry RPC@1278819886) -> (continuation of Deferred@1022259665 after retry RPC@958652460) -> (continuation of Deferred@885382485 after retry RPC@306276705), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@57cb54dc -> wakeup thread Time-limited test -> (continuation of Deferred@1213272284 after retry RPC@1169281214) -> (continuation of Deferred@698195794 after retry RPC@1217873062) -> (continuation of Deferred@1056567075 after retry RPC@1278819886) -> (continuation of Deferred@1022259665 after retry RPC@958652460) -> (continuation of Deferred@885382485 after retry RPC@306276705))) after lookup
02:03:54.288 [DEBUG - kudu-nio-1] (Connection.java:748) [peer b123247401d540528f8324235e936260(127.13.243.2:38235)] connecting to peer
02:03:54.289 [DEBUG - kudu-nio-5] (Connection.java:757) [peer b123247401d540528f8324235e936260(127.13.243.2:38235)] Successfully connected to peer
02:03:54.314 [DEBUG - kudu-nio-5] (Negotiator.java:919) Authenticated connection [id: 0xe288ccec, L:/127.0.0.1:43396 - R:/127.13.243.2:38235] using TOKEN/null
02:03:54.335 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.13.243.3:41985
02:03:54.336 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

02:03:54.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:54.337021 14284 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu
02:03:54.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu
02:03:54.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/wal
02:03:54.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/data
02:03:54.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:03:54.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:03:54.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:03:54.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:03:54.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:03:54.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:03:54.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/logs
02:03:54.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/data/info.pb
02:03:54.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:03:54.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:03:54.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:03:54.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:03:54.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:03:54.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:03:54.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:03:54.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:03:54.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:03:54.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:03:54.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.13.243.3:41985
02:03:54.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.13.243.3
02:03:54.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.13.243.62:45883,127.13.243.61:44961,127.13.243.60:35015
02:03:54.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=43625
02:03:54.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.13.243.3
02:03:54.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.13.243.20:34675
02:03:54.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:03:54.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:03:54.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:03:54.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:54.664489 15627 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:03:54.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:54.664841 15627 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:03:54.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:54.664958 15627 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:03:54.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:54.673235 15627 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:03:54.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:54.673403 15627 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.243.3
02:03:54.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:54.684001 15627 tablet_server_runner.cc:78] Tablet server non-default flags:
02:03:54.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:03:54.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.13.243.20:34675
02:03:54.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:03:54.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:03:54.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/data
02:03:54.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/wal
02:03:54.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:03:54.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:03:54.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.13.243.3:41985
02:03:54.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:03:54.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:03:54.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:03:54.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/data/info.pb
02:03:54.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.13.243.3
02:03:54.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=43625
02:03:54.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.13.243.62:45883,127.13.243.61:44961,127.13.243.60:35015
02:03:54.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:03:54.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:03:54.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:03:54.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:03:54.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:03:54.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.13.243.3
02:03:54.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/logs
02:03:54.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:03:54.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:03:54.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:03:54.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:03:54.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:03:54.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision a3f35afc2b6dad6aa403ec58378221544a26edd0
02:03:54.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:03:54.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:40:41 UTC on bdcb31816ec0
02:03:54.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 16849
02:03:54.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
02:03:54.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:54.685684 15627 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:03:54.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:54.687521 15627 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:03:54.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:54.702189 15635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:54.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:54.702201 15634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:54.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:54.704766 15637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:54.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:54.704813 15627 server_base.cc:1061] running on GCE node
02:03:54.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:54.705538 15627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:03:54.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:54.706636 15627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:03:54.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:54.707852 15627 hybrid_clock.cc:648] HybridClock initialized: now 1783476234707798 us; error 52 us; skew 500 ppm
02:03:54.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:54.708153 15627 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:03:54.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:54.710506 15627 webserver.cc:534] Webserver started at http://127.13.243.3:43625/ using document root <none> and password file <none>
02:03:54.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:54.711241 15627 fs_manager.cc:362] Metadata directory not provided
02:03:54.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:54.711328 15627 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:03:54.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:54.716806 15627 fs_manager.cc:714] Time spent opening directory manager: real 0.004s	user 0.002s	sys 0.003s
02:03:54.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:54.720265 15643 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:03:54.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:54.721715 15627 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
02:03:54.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:54.721848 15627 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/data,/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/wal
02:03:54.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "7f08c11f6dc94dc3aa69febd1051f2e9"
02:03:54.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:03:37 on dist-test-slave-pjt5"
02:03:54.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:54.722610 15627 fs_report.cc:389] FS layout report
02:03:54.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:03:54.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/wal
02:03:54.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/wal
02:03:54.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/data/data
02:03:54.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:03:54.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:03:54.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:03:54.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:03:54.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:03:54.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:03:54.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:03:54.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:03:54.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:03:54.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:03:54.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:03:54.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:54.738162 15627 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:03:54.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:54.739012 15627 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:03:54.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:54.739250 15627 kserver.cc:163] Server-wide thread pool size limit: 3276
02:03:54.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:54.740226 15627 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:03:54.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:54.742460 15650 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
02:03:54.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:54.748611 15627 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
02:03:54.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:54.748703 15627 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.007s	user 0.001s	sys 0.000s
02:03:54.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:54.748799 15627 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
02:03:54.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:54.751839 15627 ts_tablet_manager.cc:616] Registered 1 tablets
02:03:54.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:54.751960 15627 ts_tablet_manager.cc:595] Time spent register tablets: real 0.003s	user 0.002s	sys 0.000s
02:03:54.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:54.752391 15650 tablet_bootstrap.cc:492] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9: Bootstrap starting.
02:03:54.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:54.784356 15650 log.cc:826] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9: Log is configured to *not* fsync() on all Append() calls
02:03:54.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:54.801178 15627 rpc_server.cc:307] RPC server started. Bound to: 127.13.243.3:41985
02:03:54.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:54.801205 15757 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.243.3:41985 every 8 connection(s)
02:03:54.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:54.803026 15627 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/data/info.pb
02:03:54.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:54.804124 14284 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu as pid 15627
02:03:54.804 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:03:54.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:54.823599 15650 tablet_bootstrap.cc:492] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9: Bootstrap replayed 1/1 log segments. Stats: ops{read=16 overwritten=0 applied=16 ignored=0} inserts{seen=12 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
02:03:54.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:54.824460 15650 tablet_bootstrap.cc:492] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9: Bootstrap complete.
02:03:54.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:54.826264 15650 ts_tablet_manager.cc:1403] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9: Time spent bootstrapping tablet: real 0.074s	user 0.057s	sys 0.004s
02:03:54.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:54.833228 15650 raft_consensus.cc:359] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:03:54.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:54.834748 15650 raft_consensus.cc:740] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f08c11f6dc94dc3aa69febd1051f2e9, State: Initialized, Role: FOLLOWER
02:03:54.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:54.835518 15650 consensus_queue.cc:260] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 4.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:03:54.839 [DEBUG - kudu-nio-5] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 15 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291, tablet=89c23a195e1e46c19d1cff4420b1f6e5, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:03:54.841 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.13.243.2:38235
02:03:54.841 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

02:03:54.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:54.842234 14284 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu pid 15256
02:03:54.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:54.844060 15759 heartbeater.cc:344] Connected to a master server at 127.13.243.62:45883
02:03:54.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:54.844556 15759 heartbeater.cc:461] Registering TS with master...
02:03:54.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:54.845850 15759 heartbeater.cc:507] Master 127.13.243.62:45883 requested a full tablet report, sending...
02:03:54.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:54.848868 15650 ts_tablet_manager.cc:1434] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9: Time spent starting tablet: real 0.022s	user 0.013s	sys 0.000s
02:03:54.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:54.849283 14325 ts_manager.cc:194] Re-registered known tserver with Master: 7f08c11f6dc94dc3aa69febd1051f2e9 (127.13.243.3:41985)
02:03:54.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:54.852896 15762 heartbeater.cc:344] Connected to a master server at 127.13.243.61:44961
02:03:54.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:54.853039 15762 heartbeater.cc:461] Registering TS with master...
02:03:54.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:54.853562 15762 heartbeater.cc:507] Master 127.13.243.61:44961 requested a full tablet report, sending...
02:03:54.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:54.854956 14397 ts_manager.cc:194] Re-registered known tserver with Master: 7f08c11f6dc94dc3aa69febd1051f2e9 (127.13.243.3:41985)
02:03:54.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:54.856851 15758 heartbeater.cc:344] Connected to a master server at 127.13.243.60:35015
02:03:54.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:54.856983 15758 heartbeater.cc:461] Registering TS with master...
02:03:54.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:54.857434 15758 heartbeater.cc:507] Master 127.13.243.60:35015 requested a full tablet report, sending...
02:03:54.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:54.859033 14468 ts_manager.cc:194] Re-registered known tserver with Master: 7f08c11f6dc94dc3aa69febd1051f2e9 (127.13.243.3:41985)
02:03:54.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:54.860680 14468 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.13.243.3:56463
02:03:54.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:54.866430 15380 generic_service.cc:196] Checking for leaks (request via RPC)
02:03:54.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:54.868188 15758 heartbeater.cc:499] Master 127.13.243.60:35015 was elected leader, sending a full tablet report...
02:03:54.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:54.967306 14284 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu with pid 15256
02:03:54.985 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:03:54.985 [DEBUG - kudu-nio-5] (Connection.java:239) [peer b123247401d540528f8324235e936260(127.13.243.2:38235)] handling channelInactive
02:03:54.985 [DEBUG - kudu-nio-5] (Connection.java:709) [peer b123247401d540528f8324235e936260(127.13.243.2:38235)] cleaning up while in state READY due to: connection closed
02:03:55.005 [DEBUG - pool-2-thread-1] (Connection.java:748) [peer b123247401d540528f8324235e936260(127.13.243.2:38235)] connecting to peer
02:03:55.006 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235
02:03:55.006 [DEBUG - kudu-nio-6] (Connection.java:709) [peer b123247401d540528f8324235e936260(127.13.243.2:38235)] cleaning up while in state CONNECTING due to: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235
02:03:55.006 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location b123247401d540528f8324235e936260(127.13.243.2:38235) for tablet 89c23a195e1e46c19d1cff4420b1f6e5: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235
02:03:55.027 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:03:55.028 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421),7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985),b123247401d540528f8324235e936260(127.13.243.2:38235)[L]]}]
02:03:55.028 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=89c23a195e1e46c19d1cff4420b1f6e5, attempt=3, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=42), Traces: [0ms] sending RPC to server b123247401d540528f8324235e936260, [1ms] delaying RPC due to: Illegal state: connection is terminated, [20ms] sending RPC to server b123247401d540528f8324235e936260, [21ms] received response from server b123247401d540528f8324235e936260: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [21ms] delaying RPC due to: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [42ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=Deferred@59438400(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f87c036bed0@1a999e31 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@8b3b669 -> wakeup thread Time-limited test)) after lookup
02:03:55.029 [DEBUG - kudu-nio-1] (Connection.java:748) [peer b123247401d540528f8324235e936260(127.13.243.2:38235)] connecting to peer
02:03:55.030 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235
02:03:55.030 [DEBUG - kudu-nio-7] (Connection.java:709) [peer b123247401d540528f8324235e936260(127.13.243.2:38235)] cleaning up while in state CONNECTING due to: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235
02:03:55.030 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location b123247401d540528f8324235e936260(127.13.243.2:38235) for tablet 89c23a195e1e46c19d1cff4420b1f6e5: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235
02:03:55.047 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:03:55.047 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421),7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985),b123247401d540528f8324235e936260(127.13.243.2:38235)[L]]}]
02:03:55.047 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=89c23a195e1e46c19d1cff4420b1f6e5, attempt=5, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=61), Traces: [0ms] sending RPC to server b123247401d540528f8324235e936260, [1ms] delaying RPC due to: Illegal state: connection is terminated, [20ms] sending RPC to server b123247401d540528f8324235e936260, [21ms] received response from server b123247401d540528f8324235e936260: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [21ms] delaying RPC due to: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [42ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [43ms] sending RPC to server b123247401d540528f8324235e936260, [45ms] received response from server b123247401d540528f8324235e936260: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [45ms] delaying RPC due to: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [58ms] refreshing cache from master, [58ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [61ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=Deferred@59438400(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f87c036bed0@1a999e31 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1858101152 after retry RPC@2077601007), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@8b3b669 -> wakeup thread Time-limited test -> (continuation of Deferred@1858101152 after retry RPC@2077601007))) after lookup
02:03:55.048 [DEBUG - kudu-nio-1] (Connection.java:748) [peer b123247401d540528f8324235e936260(127.13.243.2:38235)] connecting to peer
02:03:55.048 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235
02:03:55.049 [DEBUG - kudu-nio-0] (Connection.java:709) [peer b123247401d540528f8324235e936260(127.13.243.2:38235)] cleaning up while in state CONNECTING due to: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235
02:03:55.049 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location b123247401d540528f8324235e936260(127.13.243.2:38235) for tablet 89c23a195e1e46c19d1cff4420b1f6e5: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235
02:03:55.107 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:03:55.107 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421),7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985),b123247401d540528f8324235e936260(127.13.243.2:38235)[L]]}]
02:03:55.108 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=89c23a195e1e46c19d1cff4420b1f6e5, attempt=7, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=122), Traces: [0ms] sending RPC to server b123247401d540528f8324235e936260, [1ms] delaying RPC due to: Illegal state: connection is terminated, [20ms] sending RPC to server b123247401d540528f8324235e936260, [21ms] received response from server b123247401d540528f8324235e936260: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [21ms] delaying RPC due to: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [42ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [43ms] sending RPC to server b123247401d540528f8324235e936260, [45ms] received response from server b123247401d540528f8324235e936260: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [45ms] delaying RPC due to: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [58ms] refreshing cache from master, [58ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [61ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [62ms] sending RPC to server b123247401d540528f8324235e936260, [64ms] received response from server b123247401d540528f8324235e936260: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [64ms] delaying RPC due to: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [119ms] refreshing cache from master, [119ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [122ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=Deferred@59438400(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f87c036bed0@1a999e31 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1858101152 after retry RPC@2077601007) -> (continuation of Deferred@1698927878 after retry RPC@858255253), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@8b3b669 -> wakeup thread Time-limited test -> (continuation of Deferred@1858101152 after retry RPC@2077601007) -> (continuation of Deferred@1698927878 after retry RPC@858255253))) after lookup
02:03:55.108 [DEBUG - kudu-nio-1] (Connection.java:748) [peer b123247401d540528f8324235e936260(127.13.243.2:38235)] connecting to peer
02:03:55.109 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235
02:03:55.109 [DEBUG - kudu-nio-1] (Connection.java:709) [peer b123247401d540528f8324235e936260(127.13.243.2:38235)] cleaning up while in state CONNECTING due to: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235
02:03:55.109 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location b123247401d540528f8324235e936260(127.13.243.2:38235) for tablet 89c23a195e1e46c19d1cff4420b1f6e5: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235
02:03:55.307 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:03:55.307 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421),7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985),b123247401d540528f8324235e936260(127.13.243.2:38235)[L]]}]
02:03:55.307 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=89c23a195e1e46c19d1cff4420b1f6e5, attempt=9, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=322), Traces: [0ms] sending RPC to server b123247401d540528f8324235e936260, [1ms] delaying RPC due to: Illegal state: connection is terminated, [20ms] sending RPC to server b123247401d540528f8324235e936260, [21ms] received response from server b123247401d540528f8324235e936260: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [21ms] delaying RPC due to: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [42ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [43ms] sending RPC to server b123247401d540528f8324235e936260, [45ms] received response from server b123247401d540528f8324235e936260: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [45ms] delaying RPC due to: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [58ms] refreshing cache from master, [58ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [61ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [62ms] sending RPC to server b123247401d540528f8324235e936260, [64ms] received response from server b123247401d540528f8324235e936260: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [64ms] delaying RPC due to: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [119ms] refreshing cache from master, [119ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [122ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [123ms] sending RPC to server b123247401d540528f8324235e936260, [124ms] received response from server b123247401d540528f8324235e936260: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [124ms] delaying RPC due to: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [319ms] refreshing cache from master, [319ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [321ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=Deferred@59438400(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f87c036bed0@1a999e31 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1858101152 after retry RPC@2077601007) -> (continuation of Deferred@1698927878 after retry RPC@858255253) -> (continuation of Deferred@133422762 after retry RPC@1556645910), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@8b3b669 -> wakeup thread Time-limited test -> (continuation of Deferred@1858101152 after retry RPC@2077601007) -> (continuation of Deferred@1698927878 after retry RPC@858255253) -> (continuation of Deferred@133422762 after retry RPC@1556645910))) after lookup
02:03:55.308 [DEBUG - kudu-nio-1] (Connection.java:748) [peer b123247401d540528f8324235e936260(127.13.243.2:38235)] connecting to peer
02:03:55.309 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235
02:03:55.309 [DEBUG - kudu-nio-2] (Connection.java:709) [peer b123247401d540528f8324235e936260(127.13.243.2:38235)] cleaning up while in state CONNECTING due to: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235
02:03:55.309 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location b123247401d540528f8324235e936260(127.13.243.2:38235) for tablet 89c23a195e1e46c19d1cff4420b1f6e5: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235
02:03:55.626 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:03:55.627 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421),7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985),b123247401d540528f8324235e936260(127.13.243.2:38235)[L]]}]
02:03:55.627 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=89c23a195e1e46c19d1cff4420b1f6e5, attempt=11, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=641), Traces: [0ms] sending RPC to server b123247401d540528f8324235e936260, [1ms] delaying RPC due to: Illegal state: connection is terminated, [20ms] sending RPC to server b123247401d540528f8324235e936260, [21ms] received response from server b123247401d540528f8324235e936260: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [21ms] delaying RPC due to: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [42ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [43ms] sending RPC to server b123247401d540528f8324235e936260, [45ms] received response from server b123247401d540528f8324235e936260: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [45ms] delaying RPC due to: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [58ms] refreshing cache from master, [58ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [61ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [62ms] sending RPC to server b123247401d540528f8324235e936260, [64ms] received response from server b123247401d540528f8324235e936260: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [64ms] delaying RPC due to: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [119ms] refreshing cache from master, [119ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [122ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [123ms] sending RPC to server b123247401d540528f8324235e936260, [124ms] received response from server b123247401d540528f8324235e936260: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [124ms] delaying RPC due to: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [319ms] refreshing cache from master, [319ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [321ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [323ms] sending RPC to server b123247401d540528f8324235e936260, [324ms] received response from server b123247401d540528f8324235e936260: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [324ms] delaying RPC due to: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [638ms] refreshing cache from master, [639ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [641ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=Deferred@59438400(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f87c036bed0@1a999e31 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1858101152 after retry RPC@2077601007) -> (continuation of Deferred@1698927878 after retry RPC@858255253) -> (continuation of Deferred@133422762 after retry RPC@1556645910) -> (continuation of Deferred@1900837735 after retry RPC@1776080224), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@8b3b669 -> wakeup thread Time-limited test -> (continuation of Deferred@1858101152 after retry RPC@2077601007) -> (continuation of Deferred@1698927878 after retry RPC@858255253) -> (continuation of Deferred@133422762 after retry RPC@1556645910) -> (continuation of Deferred@1900837735 after retry RPC@1776080224))) after lookup
02:03:55.628 [DEBUG - kudu-nio-1] (Connection.java:748) [peer b123247401d540528f8324235e936260(127.13.243.2:38235)] connecting to peer
02:03:55.629 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235
02:03:55.629 [DEBUG - kudu-nio-3] (Connection.java:709) [peer b123247401d540528f8324235e936260(127.13.243.2:38235)] cleaning up while in state CONNECTING due to: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235
02:03:55.629 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location b123247401d540528f8324235e936260(127.13.243.2:38235) for tablet 89c23a195e1e46c19d1cff4420b1f6e5: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235
02:03:56.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:56.255198 15773 raft_consensus.cc:493] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b123247401d540528f8324235e936260)
02:03:56.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:56.255535 15773 raft_consensus.cc:515] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:03:56.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:56.256291 15773 leader_election.cc:290] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b123247401d540528f8324235e936260 (127.13.243.2:38235), 7f08c11f6dc94dc3aa69febd1051f2e9 (127.13.243.3:41985)
02:03:56.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:56.260985 15466 leader_election.cc:336] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer b123247401d540528f8324235e936260 (127.13.243.2:38235): Network error: Client connection negotiation failed: client connection to 127.13.243.2:38235: connect: Connection refused (error 111)
02:03:56.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:56.266245 15712 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89c23a195e1e46c19d1cff4420b1f6e5" candidate_uuid: "6422ba2afbf742febb59b258c6ac5101" candidate_term: 6 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" is_pre_election: true
02:03:56.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:56.266732 15712 raft_consensus.cc:2468] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6422ba2afbf742febb59b258c6ac5101 in term 4.
02:03:56.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:56.267573 15464 leader_election.cc:304] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6422ba2afbf742febb59b258c6ac5101, 7f08c11f6dc94dc3aa69febd1051f2e9; no voters: b123247401d540528f8324235e936260
02:03:56.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:56.267855 15773 raft_consensus.cc:2804] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [term 5 FOLLOWER]: Leader pre-election won for term 6
02:03:56.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:56.267975 15773 raft_consensus.cc:493] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [term 5 FOLLOWER]: Starting leader election (detected failure of leader b123247401d540528f8324235e936260)
02:03:56.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:56.268054 15773 raft_consensus.cc:3060] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [term 5 FOLLOWER]: Advancing to term 6
02:03:56.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:56.269974 15773 raft_consensus.cc:515] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:03:56.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:56.270653 15773 leader_election.cc:290] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [CANDIDATE]: Term 6 election: Requested vote from peers b123247401d540528f8324235e936260 (127.13.243.2:38235), 7f08c11f6dc94dc3aa69febd1051f2e9 (127.13.243.3:41985)
02:03:56.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:56.271184 15712 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89c23a195e1e46c19d1cff4420b1f6e5" candidate_uuid: "6422ba2afbf742febb59b258c6ac5101" candidate_term: 6 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9"
02:03:56.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:56.271376 15712 raft_consensus.cc:3060] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [term 4 FOLLOWER]: Advancing to term 6
02:03:56.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:56.272078 15466 leader_election.cc:336] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer b123247401d540528f8324235e936260 (127.13.243.2:38235): Network error: Client connection negotiation failed: client connection to 127.13.243.2:38235: connect: Connection refused (error 111)
02:03:56.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:56.275265 15712 raft_consensus.cc:2468] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6422ba2afbf742febb59b258c6ac5101 in term 6.
02:03:56.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:56.275658 15464 leader_election.cc:304] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6422ba2afbf742febb59b258c6ac5101, 7f08c11f6dc94dc3aa69febd1051f2e9; no voters: b123247401d540528f8324235e936260
02:03:56.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:56.275931 15773 raft_consensus.cc:2804] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [term 6 FOLLOWER]: Leader election won for term 6
02:03:56.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:56.276342 15773 raft_consensus.cc:697] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [term 6 LEADER]: Becoming Leader. State: Replica: 6422ba2afbf742febb59b258c6ac5101, State: Running, Role: LEADER
02:03:56.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:56.276858 15773 consensus_queue.cc:237] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 5.20, Last appended by leader: 20, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:03:56.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:56.280498 14468 catalog_manager.cc:5696] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 reported cstate change: term changed from 5 to 6, leader changed from b123247401d540528f8324235e936260 (127.13.243.2) to 6422ba2afbf742febb59b258c6ac5101 (127.13.243.1). New cstate: current_term: 6 leader_uuid: "6422ba2afbf742febb59b258c6ac5101" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } health_report { overall_health: UNKNOWN } } }
02:03:56.667 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:03:56.667 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)[L],7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985),b123247401d540528f8324235e936260(127.13.243.2:38235)]}]
02:03:56.667 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=89c23a195e1e46c19d1cff4420b1f6e5, attempt=13, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=1681), Traces: [0ms] sending RPC to server b123247401d540528f8324235e936260, [1ms] delaying RPC due to: Illegal state: connection is terminated, [20ms] sending RPC to server b123247401d540528f8324235e936260, [21ms] received response from server b123247401d540528f8324235e936260: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [21ms] delaying RPC due to: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [42ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [43ms] sending RPC to server b123247401d540528f8324235e936260, [45ms] received response from server b123247401d540528f8324235e936260: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [45ms] delaying RPC due to: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [58ms] refreshing cache from master, [58ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [61ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [62ms] sending RPC to server b123247401d540528f8324235e936260, [64ms] received response from server b123247401d540528f8324235e936260: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [64ms] delaying RPC due to: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [119ms] refreshing cache from master, [119ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [122ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [123ms] sending RPC to server b123247401d540528f8324235e936260, [124ms] received response from server b123247401d540528f8324235e936260: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [124ms] delaying RPC due to: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [319ms] refreshing cache from master, [319ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [321ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [323ms] sending RPC to server b123247401d540528f8324235e936260, [324ms] received response from server b123247401d540528f8324235e936260: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [324ms] delaying RPC due to: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [638ms] refreshing cache from master, [639ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [641ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [643ms] sending RPC to server b123247401d540528f8324235e936260, [644ms] received response from server b123247401d540528f8324235e936260: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [644ms] delaying RPC due to: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [1679ms] refreshing cache from master, [1679ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [1681ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=Deferred@59438400(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f87c036bed0@1a999e31 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1858101152 after retry RPC@2077601007) -> (continuation of Deferred@1698927878 after retry RPC@858255253) -> (continuation of Deferred@133422762 after retry RPC@1556645910) -> (continuation of Deferred@1900837735 after retry RPC@1776080224) -> (continuation of Deferred@574737115 after retry RPC@758737958), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@8b3b669 -> wakeup thread Time-limited test -> (continuation of Deferred@1858101152 after retry RPC@2077601007) -> (continuation of Deferred@1698927878 after retry RPC@858255253) -> (continuation of Deferred@133422762 after retry RPC@1556645910) -> (continuation of Deferred@1900837735 after retry RPC@1776080224) -> (continuation of Deferred@574737115 after retry RPC@758737958))) after lookup
02:03:56.668 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)] connecting to peer
02:03:56.669 [DEBUG - kudu-nio-4] (Connection.java:757) [peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)] Successfully connected to peer
02:03:56.693 [DEBUG - kudu-nio-4] (Negotiator.java:919) Authenticated connection [id: 0xe2dff288, L:/127.0.0.1:43224 - R:/127.13.243.1:33421] using TOKEN/null
02:03:56.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:56.699563 15712 raft_consensus.cc:1275] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [term 6 FOLLOWER]: Refusing update from remote peer 6422ba2afbf742febb59b258c6ac5101: Log matching property violated. Preceding OpId in replica: term: 4 index: 16. Preceding OpId from leader: term: 6 index: 22. (index mismatch)
02:03:56.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:56.699949 15466 consensus_peers.cc:597] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 -> Peer b123247401d540528f8324235e936260 (127.13.243.2:38235): Couldn't send request to peer b123247401d540528f8324235e936260. Status: Network error: Client connection negotiation failed: client connection to 127.13.243.2:38235: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
02:03:56.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:56.700533 15773 consensus_queue.cc:1048] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 16, Time since last communication: 0.000s
02:03:56.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:56.708256 15793 mvcc.cc:204] Tried to move back new op lower bound from 7305118655765250048 to 7305118653314191360. Current Snapshot: MvccSnapshot[applied={T|T < 7305118655765250048}]
02:03:56.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:56.713577 15791 mvcc.cc:204] Tried to move back new op lower bound from 7305118665510748160 to 7305118663793664000. Current Snapshot: MvccSnapshot[applied={T|T < 7305118665510748160}]
02:03:56.734 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.13.243.2:38235
02:03:56.736 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

02:03:56.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:56.737207 14284 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu
02:03:56.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu
02:03:56.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-1/wal
02:03:56.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-1/data
02:03:56.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:03:56.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:03:56.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:03:56.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:03:56.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:03:56.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:03:56.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-1/logs
02:03:56.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-1/data/info.pb
02:03:56.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:03:56.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:03:56.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:03:56.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:03:56.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:03:56.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:03:56.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:03:56.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:03:56.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:03:56.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:03:56.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.13.243.2:38235
02:03:56.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.13.243.2
02:03:56.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.13.243.62:45883,127.13.243.61:44961,127.13.243.60:35015
02:03:56.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42383
02:03:56.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.13.243.2
02:03:56.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.13.243.20:34675
02:03:56.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:03:56.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:03:56.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:03:57.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:57.071889 15800 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:03:57.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:57.072258 15800 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:03:57.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:57.072376 15800 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:03:57.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:57.080904 15800 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:03:57.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:57.081161 15800 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.243.2
02:03:57.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:57.091841 15800 tablet_server_runner.cc:78] Tablet server non-default flags:
02:03:57.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:03:57.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.13.243.20:34675
02:03:57.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:03:57.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:03:57.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-1/data
02:03:57.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-1/wal
02:03:57.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:03:57.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:03:57.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.13.243.2:38235
02:03:57.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:03:57.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:03:57.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:03:57.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-1/data/info.pb
02:03:57.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.13.243.2
02:03:57.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42383
02:03:57.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.13.243.62:45883,127.13.243.61:44961,127.13.243.60:35015
02:03:57.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:03:57.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:03:57.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:03:57.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:03:57.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:03:57.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.13.243.2
02:03:57.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-1/logs
02:03:57.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:03:57.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:03:57.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:03:57.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:03:57.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:03:57.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision a3f35afc2b6dad6aa403ec58378221544a26edd0
02:03:57.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:03:57.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:40:41 UTC on bdcb31816ec0
02:03:57.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 16849
02:03:57.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
02:03:57.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:57.094054 15800 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:03:57.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:57.096052 15800 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:03:57.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:57.106436 15806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:57.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:57.106456 15807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:57.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:57.107384 15809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:57.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:57.108739 15800 server_base.cc:1061] running on GCE node
02:03:57.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:57.109333 15800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:03:57.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:57.110385 15800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:03:57.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:57.111616 15800 hybrid_clock.cc:648] HybridClock initialized: now 1783476237111546 us; error 61 us; skew 500 ppm
02:03:57.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:57.111902 15800 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:03:57.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:57.114188 15800 webserver.cc:534] Webserver started at http://127.13.243.2:42383/ using document root <none> and password file <none>
02:03:57.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:57.114982 15800 fs_manager.cc:362] Metadata directory not provided
02:03:57.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:57.115085 15800 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:03:57.120 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:57.120602 15800 fs_manager.cc:714] Time spent opening directory manager: real 0.004s	user 0.005s	sys 0.000s
02:03:57.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:57.123565 15815 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:03:57.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:57.125277 15800 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
02:03:57.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:57.125461 15800 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-1/data,/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-1/wal
02:03:57.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "b123247401d540528f8324235e936260"
02:03:57.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:03:37 on dist-test-slave-pjt5"
02:03:57.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:57.126449 15800 fs_report.cc:389] FS layout report
02:03:57.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:03:57.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-1/wal
02:03:57.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-1/wal
02:03:57.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-1/data/data
02:03:57.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:03:57.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:03:57.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:03:57.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:03:57.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:03:57.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:03:57.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:03:57.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:03:57.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:03:57.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:03:57.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:03:57.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:57.169736 15800 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:03:57.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:57.170681 15800 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:03:57.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:57.170989 15800 kserver.cc:163] Server-wide thread pool size limit: 3276
02:03:57.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:57.172223 15800 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:03:57.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:57.174854 15822 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
02:03:57.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:57.180691 15800 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
02:03:57.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:57.180791 15800 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.006s	user 0.000s	sys 0.000s
02:03:57.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:57.180922 15800 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
02:03:57.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:57.185276 15800 ts_tablet_manager.cc:616] Registered 1 tablets
02:03:57.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:57.185379 15800 ts_tablet_manager.cc:595] Time spent register tablets: real 0.004s	user 0.000s	sys 0.002s
02:03:57.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:57.185931 15822 tablet_bootstrap.cc:492] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260: Bootstrap starting.
02:03:57.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:57.214192 15822 log.cc:826] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260: Log is configured to *not* fsync() on all Append() calls
02:03:57.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:57.227412 15800 rpc_server.cc:307] RPC server started. Bound to: 127.13.243.2:38235
02:03:57.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:57.227622 15929 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.243.2:38235 every 8 connection(s)
02:03:57.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:57.229242 15800 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-1/data/info.pb
02:03:57.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:57.235927 14284 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu as pid 15800
02:03:57.236 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:03:57.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:57.250654 15822 tablet_bootstrap.cc:492] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260: Bootstrap replayed 1/1 log segments. Stats: ops{read=20 overwritten=0 applied=20 ignored=0} inserts{seen=15 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
02:03:57.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:57.252491 15822 tablet_bootstrap.cc:492] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260: Bootstrap complete.
02:03:57.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:57.255322 15822 ts_tablet_manager.cc:1403] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260: Time spent bootstrapping tablet: real 0.070s	user 0.042s	sys 0.024s
02:03:57.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:57.265743 15822 raft_consensus.cc:359] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:03:57.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:57.267124 15822 raft_consensus.cc:740] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: b123247401d540528f8324235e936260, State: Initialized, Role: FOLLOWER
02:03:57.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:57.268041 15822 consensus_queue.cc:260] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 20, Last appended: 5.20, Last appended by leader: 20, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:03:57.269 [DEBUG - kudu-nio-4] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 18 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291, tablet=89c23a195e1e46c19d1cff4420b1f6e5, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:03:57.270 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.13.243.1:33421
02:03:57.270 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

02:03:57.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:57.271122 14284 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu pid 15449
02:03:57.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:57.285207 15822 ts_tablet_manager.cc:1434] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260: Time spent starting tablet: real 0.030s	user 0.015s	sys 0.005s
02:03:57.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:57.298882 15575 generic_service.cc:196] Checking for leaks (request via RPC)
02:03:57.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:57.299672 15931 heartbeater.cc:344] Connected to a master server at 127.13.243.62:45883
02:03:57.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:57.300006 15931 heartbeater.cc:461] Registering TS with master...
02:03:57.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:57.301019 15931 heartbeater.cc:507] Master 127.13.243.62:45883 requested a full tablet report, sending...
02:03:57.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:57.301609 15932 heartbeater.cc:344] Connected to a master server at 127.13.243.61:44961
02:03:57.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:57.301821 15932 heartbeater.cc:461] Registering TS with master...
02:03:57.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:57.302353 15932 heartbeater.cc:507] Master 127.13.243.61:44961 requested a full tablet report, sending...
02:03:57.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:57.302460 15930 heartbeater.cc:344] Connected to a master server at 127.13.243.60:35015
02:03:57.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:57.302639 15930 heartbeater.cc:461] Registering TS with master...
02:03:57.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:57.303296 15930 heartbeater.cc:507] Master 127.13.243.60:35015 requested a full tablet report, sending...
02:03:57.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:57.304378 14397 ts_manager.cc:194] Re-registered known tserver with Master: b123247401d540528f8324235e936260 (127.13.243.2:38235)
02:03:57.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:57.305428 14468 ts_manager.cc:194] Re-registered known tserver with Master: b123247401d540528f8324235e936260 (127.13.243.2:38235)
02:03:57.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:57.306872 14468 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.13.243.2:46335
02:03:57.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:57.307299 14325 ts_manager.cc:194] Re-registered known tserver with Master: b123247401d540528f8324235e936260 (127.13.243.2:38235)
02:03:57.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:57.310392 15930 heartbeater.cc:499] Master 127.13.243.60:35015 was elected leader, sending a full tablet report...
02:03:57.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:57.404039 14284 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu with pid 15449
02:03:57.423 [DEBUG - kudu-nio-4] (Connection.java:239) [peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)] handling channelInactive
02:03:57.423 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:03:57.423 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)] cleaning up while in state READY due to: connection closed
02:03:57.424 [DEBUG - Time-limited test] (Connection.java:748) [peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)] connecting to peer
02:03:57.425 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421
02:03:57.425 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)] cleaning up while in state CONNECTING due to: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421
02:03:57.425 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421) for tablet 89c23a195e1e46c19d1cff4420b1f6e5: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421
02:03:57.447 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:03:57.447 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)[L],7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985),b123247401d540528f8324235e936260(127.13.243.2:38235)]}]
02:03:57.447 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=89c23a195e1e46c19d1cff4420b1f6e5, attempt=2, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=24), Traces: [0ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [2ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [2ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [23ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=Deferred@1114580896(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f87c036bed0@24d13b97 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@664d8252 -> wakeup thread Time-limited test)) after lookup
02:03:57.448 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)] connecting to peer
02:03:57.448 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421
02:03:57.449 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)] cleaning up while in state CONNECTING due to: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421
02:03:57.449 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421) for tablet 89c23a195e1e46c19d1cff4420b1f6e5: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421
02:03:57.467 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:03:57.467 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)[L],7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985),b123247401d540528f8324235e936260(127.13.243.2:38235)]}]
02:03:57.467 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=89c23a195e1e46c19d1cff4420b1f6e5, attempt=4, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=44), Traces: [0ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [2ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [2ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [23ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [24ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [26ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [26ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [41ms] refreshing cache from master, [41ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [44ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=Deferred@1114580896(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f87c036bed0@24d13b97 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1071836401 after retry RPC@901228109), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@664d8252 -> wakeup thread Time-limited test -> (continuation of Deferred@1071836401 after retry RPC@901228109))) after lookup
02:03:57.468 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)] connecting to peer
02:03:57.469 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421
02:03:57.469 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)] cleaning up while in state CONNECTING due to: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421
02:03:57.469 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421) for tablet 89c23a195e1e46c19d1cff4420b1f6e5: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421
02:03:57.507 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:03:57.507 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)[L],7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985),b123247401d540528f8324235e936260(127.13.243.2:38235)]}]
02:03:57.507 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=89c23a195e1e46c19d1cff4420b1f6e5, attempt=6, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=84), Traces: [0ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [2ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [2ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [23ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [24ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [26ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [26ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [41ms] refreshing cache from master, [41ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [44ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [45ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [46ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [46ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [81ms] refreshing cache from master, [81ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [84ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=Deferred@1114580896(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f87c036bed0@24d13b97 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1071836401 after retry RPC@901228109) -> (continuation of Deferred@1837554964 after retry RPC@1703643943), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@664d8252 -> wakeup thread Time-limited test -> (continuation of Deferred@1071836401 after retry RPC@901228109) -> (continuation of Deferred@1837554964 after retry RPC@1703643943))) after lookup
02:03:57.508 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)] connecting to peer
02:03:57.509 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421
02:03:57.509 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)] cleaning up while in state CONNECTING due to: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421
02:03:57.509 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421) for tablet 89c23a195e1e46c19d1cff4420b1f6e5: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421
02:03:57.627 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:03:57.627 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)[L],7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985),b123247401d540528f8324235e936260(127.13.243.2:38235)]}]
02:03:57.627 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=89c23a195e1e46c19d1cff4420b1f6e5, attempt=8, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=204), Traces: [0ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [2ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [2ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [23ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [24ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [26ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [26ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [41ms] refreshing cache from master, [41ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [44ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [45ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [46ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [46ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [81ms] refreshing cache from master, [81ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [84ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [85ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [86ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [86ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [200ms] refreshing cache from master, [201ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [203ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=Deferred@1114580896(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f87c036bed0@24d13b97 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1071836401 after retry RPC@901228109) -> (continuation of Deferred@1837554964 after retry RPC@1703643943) -> (continuation of Deferred@158756674 after retry RPC@815599608), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@664d8252 -> wakeup thread Time-limited test -> (continuation of Deferred@1071836401 after retry RPC@901228109) -> (continuation of Deferred@1837554964 after retry RPC@1703643943) -> (continuation of Deferred@158756674 after retry RPC@815599608))) after lookup
02:03:57.628 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)] connecting to peer
02:03:57.629 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421
02:03:57.629 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)] cleaning up while in state CONNECTING due to: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421
02:03:57.629 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421) for tablet 89c23a195e1e46c19d1cff4420b1f6e5: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421
02:03:57.888 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:03:57.888 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)[L],7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985),b123247401d540528f8324235e936260(127.13.243.2:38235)]}]
02:03:57.888 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=89c23a195e1e46c19d1cff4420b1f6e5, attempt=10, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=465), Traces: [0ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [2ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [2ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [23ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [24ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [26ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [26ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [41ms] refreshing cache from master, [41ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [44ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [45ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [46ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [46ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [81ms] refreshing cache from master, [81ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [84ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [85ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [86ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [86ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [200ms] refreshing cache from master, [201ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [203ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [205ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [206ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [206ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [461ms] refreshing cache from master, [461ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [464ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=Deferred@1114580896(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f87c036bed0@24d13b97 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1071836401 after retry RPC@901228109) -> (continuation of Deferred@1837554964 after retry RPC@1703643943) -> (continuation of Deferred@158756674 after retry RPC@815599608) -> (continuation of Deferred@614557062 after retry RPC@2131590045), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@664d8252 -> wakeup thread Time-limited test -> (continuation of Deferred@1071836401 after retry RPC@901228109) -> (continuation of Deferred@1837554964 after retry RPC@1703643943) -> (continuation of Deferred@158756674 after retry RPC@815599608) -> (continuation of Deferred@614557062 after retry RPC@2131590045))) after lookup
02:03:57.889 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)] connecting to peer
02:03:57.890 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421
02:03:57.891 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)] cleaning up while in state CONNECTING due to: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421
02:03:57.891 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421) for tablet 89c23a195e1e46c19d1cff4420b1f6e5: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421
02:03:58.008 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:03:58.008 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)[L],7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985),b123247401d540528f8324235e936260(127.13.243.2:38235)]}]
02:03:58.009 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=89c23a195e1e46c19d1cff4420b1f6e5, attempt=12, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=585), Traces: [0ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [2ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [2ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [23ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [24ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [26ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [26ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [41ms] refreshing cache from master, [41ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [44ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [45ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [46ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [46ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [81ms] refreshing cache from master, [81ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [84ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [85ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [86ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [86ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [200ms] refreshing cache from master, [201ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [203ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [205ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [206ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [206ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [461ms] refreshing cache from master, [461ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [464ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [466ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [468ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [468ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [581ms] refreshing cache from master, [582ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [585ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=Deferred@1114580896(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f87c036bed0@24d13b97 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1071836401 after retry RPC@901228109) -> (continuation of Deferred@1837554964 after retry RPC@1703643943) -> (continuation of Deferred@158756674 after retry RPC@815599608) -> (continuation of Deferred@614557062 after retry RPC@2131590045) -> (continuation of Deferred@1507027721 after retry RPC@1293300285), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@664d8252 -> wakeup thread Time-limited test -> (continuation of Deferred@1071836401 after retry RPC@901228109) -> (continuation of Deferred@1837554964 after retry RPC@1703643943) -> (continuation of Deferred@158756674 after retry RPC@815599608) -> (continuation of Deferred@614557062 after retry RPC@2131590045) -> (continuation of Deferred@1507027721 after retry RPC@1293300285))) after lookup
02:03:58.009 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)] connecting to peer
02:03:58.011 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421
02:03:58.011 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)] cleaning up while in state CONNECTING due to: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421
02:03:58.011 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421) for tablet 89c23a195e1e46c19d1cff4420b1f6e5: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421
02:03:58.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:58.751107 15945 raft_consensus.cc:493] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6422ba2afbf742febb59b258c6ac5101)
02:03:58.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:58.751310 15945 raft_consensus.cc:515] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:03:58.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:58.752936 15945 leader_election.cc:290] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6422ba2afbf742febb59b258c6ac5101 (127.13.243.1:33421), b123247401d540528f8324235e936260 (127.13.243.2:38235)
02:03:58.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:58.754233 15644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.243.1:33421: connect: Connection refused (error 111)
02:03:58.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:58.755745 15644 leader_election.cc:336] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 6422ba2afbf742febb59b258c6ac5101 (127.13.243.1:33421): Network error: Client connection negotiation failed: client connection to 127.13.243.1:33421: connect: Connection refused (error 111)
02:03:58.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:58.762116 15884 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89c23a195e1e46c19d1cff4420b1f6e5" candidate_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" candidate_term: 7 candidate_status { last_received { term: 6 index: 24 } } ignore_live_leader: false dest_uuid: "b123247401d540528f8324235e936260" is_pre_election: true
02:03:58.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:58.762636 15884 raft_consensus.cc:2468] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f08c11f6dc94dc3aa69febd1051f2e9 in term 5.
02:03:58.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:58.763597 15646 leader_election.cc:304] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7f08c11f6dc94dc3aa69febd1051f2e9, b123247401d540528f8324235e936260; no voters: 6422ba2afbf742febb59b258c6ac5101
02:03:58.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:58.764029 15945 raft_consensus.cc:2804] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [term 6 FOLLOWER]: Leader pre-election won for term 7
02:03:58.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:58.764176 15945 raft_consensus.cc:493] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 6422ba2afbf742febb59b258c6ac5101)
02:03:58.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:58.764266 15945 raft_consensus.cc:3060] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [term 6 FOLLOWER]: Advancing to term 7
02:03:58.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:58.766619 15945 raft_consensus.cc:515] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:03:58.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:58.767145 15945 leader_election.cc:290] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [CANDIDATE]: Term 7 election: Requested vote from peers 6422ba2afbf742febb59b258c6ac5101 (127.13.243.1:33421), b123247401d540528f8324235e936260 (127.13.243.2:38235)
02:03:58.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:58.767733 15884 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89c23a195e1e46c19d1cff4420b1f6e5" candidate_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" candidate_term: 7 candidate_status { last_received { term: 6 index: 24 } } ignore_live_leader: false dest_uuid: "b123247401d540528f8324235e936260"
02:03:58.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:58.767952 15884 raft_consensus.cc:3060] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [term 5 FOLLOWER]: Advancing to term 7
02:03:58.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:58.768813 15644 leader_election.cc:336] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 6422ba2afbf742febb59b258c6ac5101 (127.13.243.1:33421): Network error: Client connection negotiation failed: client connection to 127.13.243.1:33421: connect: Connection refused (error 111)
02:03:58.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:58.771742 15884 raft_consensus.cc:2468] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f08c11f6dc94dc3aa69febd1051f2e9 in term 7.
02:03:58.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:58.772305 15646 leader_election.cc:304] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7f08c11f6dc94dc3aa69febd1051f2e9, b123247401d540528f8324235e936260; no voters: 6422ba2afbf742febb59b258c6ac5101
02:03:58.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:58.772629 15945 raft_consensus.cc:2804] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [term 7 FOLLOWER]: Leader election won for term 7
02:03:58.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:58.773124 15945 raft_consensus.cc:697] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [term 7 LEADER]: Becoming Leader. State: Replica: 7f08c11f6dc94dc3aa69febd1051f2e9, State: Running, Role: LEADER
02:03:58.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:58.773748 15945 consensus_queue.cc:237] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 6.24, Last appended by leader: 24, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:03:58.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:58.777809 14468 catalog_manager.cc:5696] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 reported cstate change: term changed from 6 to 7, leader changed from 6422ba2afbf742febb59b258c6ac5101 (127.13.243.1) to 7f08c11f6dc94dc3aa69febd1051f2e9 (127.13.243.3). New cstate: current_term: 7 leader_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } health_report { overall_health: HEALTHY } } }
02:03:58.790 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:03:58.790 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)[L],7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985),b123247401d540528f8324235e936260(127.13.243.2:38235)]}]
02:03:58.791 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=89c23a195e1e46c19d1cff4420b1f6e5, attempt=14, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=1367), Traces: [0ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [2ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [2ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [23ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [24ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [26ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [26ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [41ms] refreshing cache from master, [41ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [44ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [45ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [46ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [46ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [81ms] refreshing cache from master, [81ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [84ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [85ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [86ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [86ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [200ms] refreshing cache from master, [201ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [203ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [205ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [206ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [206ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [461ms] refreshing cache from master, [461ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [464ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [466ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [468ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [468ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [581ms] refreshing cache from master, [582ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [585ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [586ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [588ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [588ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [1361ms] refreshing cache from master, [1361ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [1366ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=Deferred@1114580896(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f87c036bed0@24d13b97 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1071836401 after retry RPC@901228109) -> (continuation of Deferred@1837554964 after retry RPC@1703643943) -> (continuation of Deferred@158756674 after retry RPC@815599608) -> (continuation of Deferred@614557062 after retry RPC@2131590045) -> (continuation of Deferred@1507027721 after retry RPC@1293300285) -> (continuation of Deferred@129714232 after retry RPC@2067797288), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@664d8252 -> wakeup thread Time-limited test -> (continuation of Deferred@1071836401 after retry RPC@901228109) -> (continuation of Deferred@1837554964 after retry RPC@1703643943) -> (continuation of Deferred@158756674 after retry RPC@815599608) -> (continuation of Deferred@614557062 after retry RPC@2131590045) -> (continuation of Deferred@1507027721 after retry RPC@1293300285) -> (continuation of Deferred@129714232 after retry RPC@2067797288))) after lookup
02:03:58.794 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)] connecting to peer
02:03:58.795 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421
02:03:58.795 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)] cleaning up while in state CONNECTING due to: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421
02:03:58.795 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421) for tablet 89c23a195e1e46c19d1cff4420b1f6e5: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421
02:03:59.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:59.196204 15884 raft_consensus.cc:1275] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [term 7 FOLLOWER]: Refusing update from remote peer 7f08c11f6dc94dc3aa69febd1051f2e9: Log matching property violated. Preceding OpId in replica: term: 5 index: 20. Preceding OpId from leader: term: 7 index: 25. (index mismatch)
02:03:59.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:03:59.197278 15945 consensus_queue.cc:1048] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 20, Time since last communication: 0.000s
02:03:59.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:03:59.212634 15644 consensus_peers.cc:597] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 -> Peer 6422ba2afbf742febb59b258c6ac5101 (127.13.243.1:33421): Couldn't send request to peer 6422ba2afbf742febb59b258c6ac5101. Status: Network error: Client connection negotiation failed: client connection to 127.13.243.1:33421: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
02:04:01.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:01.818859 15644 consensus_peers.cc:597] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 -> Peer 6422ba2afbf742febb59b258c6ac5101 (127.13.243.1:33421): Couldn't send request to peer 6422ba2afbf742febb59b258c6ac5101. Status: Network error: Client connection negotiation failed: client connection to 127.13.243.1:33421: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
02:04:02.087 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:04:02.088 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421),7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)[L],b123247401d540528f8324235e936260(127.13.243.2:38235)]}]
02:04:02.088 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=89c23a195e1e46c19d1cff4420b1f6e5, attempt=16, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=4665), Traces: [0ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [2ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [2ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [23ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [24ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [26ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [26ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [41ms] refreshing cache from master, [41ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [44ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [45ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [46ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [46ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [81ms] refreshing cache from master, [81ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [84ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [85ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [86ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [86ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [200ms] refreshing cache from master, [201ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [203ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [205ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [206ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [206ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [461ms] refreshing cache from master, [461ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [464ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [466ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [468ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [468ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [581ms] refreshing cache from master, [582ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [585ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [586ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [588ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [588ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [1361ms] refreshing cache from master, [1361ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [1366ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [1369ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [1372ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [1372ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [4661ms] refreshing cache from master, [4661ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [4664ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=Deferred@1114580896(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f87c036bed0@24d13b97 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1071836401 after retry RPC@901228109) -> (continuation of Deferred@1837554964 after retry RPC@1703643943) -> (continuation of Deferred@158756674 after retry RPC@815599608) -> (continuation of Deferred@614557062 after retry RPC@2131590045) -> (continuation of Deferred@1507027721 after retry RPC@1293300285) -> (continuation of Deferred@129714232 after retry RPC@2067797288) -> (continuation of Deferred@1240514860 after retry RPC@845313135), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@664d8252 -> wakeup thread Time-limited test -> (continuation of Deferred@1071836401 after retry RPC@901228109) -> (continuation of Deferred@1837554964 after retry RPC@1703643943) -> (continuation of Deferred@158756674 after retry RPC@815599608) -> (continuation of Deferred@614557062 after retry RPC@2131590045) -> (continuation of Deferred@1507027721 after retry RPC@1293300285) -> (continuation of Deferred@129714232 after retry RPC@2067797288) -> (continuation of Deferred@1240514860 after retry RPC@845313135))) after lookup
02:04:02.089 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)] connecting to peer
02:04:02.090 [DEBUG - kudu-nio-5] (Connection.java:757) [peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)] Successfully connected to peer
02:04:02.110 [DEBUG - kudu-nio-5] (Negotiator.java:919) Authenticated connection [id: 0x0eb2d5d2, L:/127.0.0.1:60186 - R:/127.13.243.3:41985] using TOKEN/null
02:04:02.138 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.13.243.1:33421
02:04:02.138 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

02:04:02.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:02.139540 14284 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu
02:04:02.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu
02:04:02.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-0/wal
02:04:02.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-0/data
02:04:02.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:04:02.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:04:02.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:04:02.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:04:02.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:04:02.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:04:02.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-0/logs
02:04:02.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-0/data/info.pb
02:04:02.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:04:02.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:04:02.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:04:02.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:04:02.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:04:02.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:04:02.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:04:02.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:04:02.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:04:02.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:04:02.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.13.243.1:33421
02:04:02.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.13.243.1
02:04:02.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.13.243.62:45883,127.13.243.61:44961,127.13.243.60:35015
02:04:02.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42315
02:04:02.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.13.243.1
02:04:02.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.13.243.20:34675
02:04:02.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:04:02.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:04:02.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:04:02.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:02.472010 15986 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:04:02.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:02.472301 15986 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:04:02.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:02.472383 15986 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:04:02.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:02.480489 15986 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:04:02.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:02.480656 15986 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.243.1
02:04:02.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:02.490780 15986 tablet_server_runner.cc:78] Tablet server non-default flags:
02:04:02.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:04:02.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.13.243.20:34675
02:04:02.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:04:02.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:04:02.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-0/data
02:04:02.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-0/wal
02:04:02.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:04:02.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:04:02.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.13.243.1:33421
02:04:02.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:04:02.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:04:02.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:04:02.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-0/data/info.pb
02:04:02.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.13.243.1
02:04:02.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42315
02:04:02.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.13.243.62:45883,127.13.243.61:44961,127.13.243.60:35015
02:04:02.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:04:02.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:04:02.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:04:02.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:04:02.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:04:02.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.13.243.1
02:04:02.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-0/logs
02:04:02.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:04:02.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:04:02.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:04:02.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:04:02.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:04:02.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision a3f35afc2b6dad6aa403ec58378221544a26edd0
02:04:02.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:04:02.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:40:41 UTC on bdcb31816ec0
02:04:02.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 16849
02:04:02.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
02:04:02.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:02.492478 15986 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:04:02.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:02.494184 15986 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:04:02.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:02.503443 15994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:04:02.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:02.503772 15996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:04:02.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:02.504189 15993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:04:02.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:02.504801 15986 server_base.cc:1061] running on GCE node
02:04:02.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:02.505336 15986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:04:02.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:02.506192 15986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:04:02.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:02.507396 15986 hybrid_clock.cc:648] HybridClock initialized: now 1783476242507343 us; error 63 us; skew 500 ppm
02:04:02.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:02.507716 15986 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:04:02.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:02.509882 15986 webserver.cc:534] Webserver started at http://127.13.243.1:42315/ using document root <none> and password file <none>
02:04:02.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:02.510653 15986 fs_manager.cc:362] Metadata directory not provided
02:04:02.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:02.510743 15986 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:04:02.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:02.516400 15986 fs_manager.cc:714] Time spent opening directory manager: real 0.004s	user 0.005s	sys 0.000s
02:04:02.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:02.519232 16002 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:04:02.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:02.520649 15986 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
02:04:02.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:02.520788 15986 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-0/data,/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-0/wal
02:04:02.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "6422ba2afbf742febb59b258c6ac5101"
02:04:02.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:03:36 on dist-test-slave-pjt5"
02:04:02.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:02.521448 15986 fs_report.cc:389] FS layout report
02:04:02.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:04:02.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-0/wal
02:04:02.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-0/wal
02:04:02.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-0/data/data
02:04:02.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:04:02.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:04:02.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:04:02.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:04:02.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:04:02.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:04:02.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:04:02.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:04:02.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:04:02.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:04:02.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:04:02.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:02.538669 15986 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:04:02.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:02.539494 15986 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:04:02.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:02.539736 15986 kserver.cc:163] Server-wide thread pool size limit: 3276
02:04:02.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:02.540629 15986 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:04:02.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:02.542807 16009 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
02:04:02.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:02.548535 15986 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
02:04:02.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:02.548624 15986 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.006s	user 0.000s	sys 0.000s
02:04:02.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:02.548722 15986 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
02:04:02.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:02.552100 15986 ts_tablet_manager.cc:616] Registered 1 tablets
02:04:02.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:02.552196 15986 ts_tablet_manager.cc:595] Time spent register tablets: real 0.003s	user 0.002s	sys 0.000s
02:04:02.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:02.552781 16009 tablet_bootstrap.cc:492] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101: Bootstrap starting.
02:04:02.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:02.580538 16009 log.cc:826] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101: Log is configured to *not* fsync() on all Append() calls
02:04:02.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:02.607460 15986 rpc_server.cc:307] RPC server started. Bound to: 127.13.243.1:33421
02:04:02.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:02.607512 16116 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.243.1:33421 every 8 connection(s)
02:04:02.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:02.609001 15986 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-0/data/info.pb
02:04:02.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:02.617094 14284 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu as pid 15986
02:04:02.618 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:04:02.636 [DEBUG - kudu-nio-5] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 21 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291, tablet=89c23a195e1e46c19d1cff4420b1f6e5, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:04:02.636 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.13.243.3:41985
02:04:02.637 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

02:04:02.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:02.637485 14284 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu pid 15627
02:04:02.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:02.647476 16118 heartbeater.cc:344] Connected to a master server at 127.13.243.62:45883
02:04:02.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:02.647791 16009 tablet_bootstrap.cc:492] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101: Bootstrap replayed 1/1 log segments. Stats: ops{read=24 overwritten=0 applied=24 ignored=0} inserts{seen=18 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
02:04:02.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:02.647938 16118 heartbeater.cc:461] Registering TS with master...
02:04:02.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:02.648581 16009 tablet_bootstrap.cc:492] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101: Bootstrap complete.
02:04:02.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:02.649101 16118 heartbeater.cc:507] Master 127.13.243.62:45883 requested a full tablet report, sending...
02:04:02.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:02.650386 16117 heartbeater.cc:344] Connected to a master server at 127.13.243.60:35015
02:04:02.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:02.650516 16117 heartbeater.cc:461] Registering TS with master...
02:04:02.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:02.650578 16009 ts_tablet_manager.cc:1403] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101: Time spent bootstrapping tablet: real 0.098s	user 0.050s	sys 0.016s
02:04:02.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:02.651075 16117 heartbeater.cc:507] Master 127.13.243.60:35015 requested a full tablet report, sending...
02:04:02.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:02.653887 14468 ts_manager.cc:194] Re-registered known tserver with Master: 6422ba2afbf742febb59b258c6ac5101 (127.13.243.1:33421)
02:04:02.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:02.655529 14468 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.13.243.1:51677
02:04:02.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:02.657598 16009 raft_consensus.cc:359] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [term 6 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:04:02.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:02.658610 16009 raft_consensus.cc:740] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6422ba2afbf742febb59b258c6ac5101, State: Initialized, Role: FOLLOWER
02:04:02.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:02.659511 16009 consensus_queue.cc:260] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 24, Last appended: 6.24, Last appended by leader: 24, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:04:02.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:02.660591 14325 ts_manager.cc:194] Re-registered known tserver with Master: 6422ba2afbf742febb59b258c6ac5101 (127.13.243.1:33421)
02:04:02.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:02.662899 16119 heartbeater.cc:344] Connected to a master server at 127.13.243.61:44961
02:04:02.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:02.663057 16119 heartbeater.cc:461] Registering TS with master...
02:04:02.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:02.663488 16119 heartbeater.cc:507] Master 127.13.243.61:44961 requested a full tablet report, sending...
02:04:02.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:02.664958 15753 generic_service.cc:196] Checking for leaks (request via RPC)
02:04:02.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:02.666671 14397 ts_manager.cc:194] Re-registered known tserver with Master: 6422ba2afbf742febb59b258c6ac5101 (127.13.243.1:33421)
02:04:02.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:02.668552 16117 heartbeater.cc:499] Master 127.13.243.60:35015 was elected leader, sending a full tablet report...
02:04:02.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:02.669409 16009 ts_tablet_manager.cc:1434] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101: Time spent starting tablet: real 0.019s	user 0.014s	sys 0.004s
02:04:02.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:02.767442 14284 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu with pid 15627
02:04:02.786 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:04:02.786 [DEBUG - kudu-nio-5] (Connection.java:239) [peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)] handling channelInactive
02:04:02.786 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)] cleaning up while in state READY due to: connection closed
02:04:02.787 [DEBUG - Time-limited test] (Connection.java:748) [peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)] connecting to peer
02:04:02.788 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:04:02.788 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:04:02.788 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985) for tablet 89c23a195e1e46c19d1cff4420b1f6e5: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:04:02.807 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:04:02.807 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421),7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)[L],b123247401d540528f8324235e936260(127.13.243.2:38235)]}]
02:04:02.807 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=89c23a195e1e46c19d1cff4420b1f6e5, attempt=2, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=20), Traces: [0ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [1ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [1ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [19ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=Deferred@116512325(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f87c036bed0@27111bf8 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4965c8b6 -> wakeup thread Time-limited test)) after lookup
02:04:02.808 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)] connecting to peer
02:04:02.809 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:04:02.809 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:04:02.809 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985) for tablet 89c23a195e1e46c19d1cff4420b1f6e5: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:04:02.827 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:04:02.828 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421),7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)[L],b123247401d540528f8324235e936260(127.13.243.2:38235)]}]
02:04:02.828 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=89c23a195e1e46c19d1cff4420b1f6e5, attempt=4, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=41), Traces: [0ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [1ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [1ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [19ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [21ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [22ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [22ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [37ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [40ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=Deferred@116512325(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f87c036bed0@27111bf8 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@457316348 after retry RPC@2080726106), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4965c8b6 -> wakeup thread Time-limited test -> (continuation of Deferred@457316348 after retry RPC@2080726106))) after lookup
02:04:02.828 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)] connecting to peer
02:04:02.829 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:04:02.829 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:04:02.830 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985) for tablet 89c23a195e1e46c19d1cff4420b1f6e5: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:04:02.867 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:04:02.868 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421),7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)[L],b123247401d540528f8324235e936260(127.13.243.2:38235)]}]
02:04:02.868 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=89c23a195e1e46c19d1cff4420b1f6e5, attempt=6, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=81), Traces: [0ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [1ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [1ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [19ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [21ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [22ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [22ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [37ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [40ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [41ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [42ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [43ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [77ms] refreshing cache from master, [77ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [80ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=Deferred@116512325(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f87c036bed0@27111bf8 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@457316348 after retry RPC@2080726106) -> (continuation of Deferred@1302608783 after retry RPC@915109346), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4965c8b6 -> wakeup thread Time-limited test -> (continuation of Deferred@457316348 after retry RPC@2080726106) -> (continuation of Deferred@1302608783 after retry RPC@915109346))) after lookup
02:04:02.868 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)] connecting to peer
02:04:02.869 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:04:02.870 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:04:02.870 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985) for tablet 89c23a195e1e46c19d1cff4420b1f6e5: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:04:02.968 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:04:02.968 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421),7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)[L],b123247401d540528f8324235e936260(127.13.243.2:38235)]}]
02:04:02.968 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=89c23a195e1e46c19d1cff4420b1f6e5, attempt=8, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=181), Traces: [0ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [1ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [1ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [19ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [21ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [22ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [22ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [37ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [40ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [41ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [42ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [43ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [77ms] refreshing cache from master, [77ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [80ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [81ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [83ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [83ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [177ms] refreshing cache from master, [177ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [180ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=Deferred@116512325(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f87c036bed0@27111bf8 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@457316348 after retry RPC@2080726106) -> (continuation of Deferred@1302608783 after retry RPC@915109346) -> (continuation of Deferred@487003981 after retry RPC@747723797), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4965c8b6 -> wakeup thread Time-limited test -> (continuation of Deferred@457316348 after retry RPC@2080726106) -> (continuation of Deferred@1302608783 after retry RPC@915109346) -> (continuation of Deferred@487003981 after retry RPC@747723797))) after lookup
02:04:02.969 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)] connecting to peer
02:04:02.970 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:04:02.970 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:04:02.970 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985) for tablet 89c23a195e1e46c19d1cff4420b1f6e5: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:04:03.107 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:04:03.107 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421),7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)[L],b123247401d540528f8324235e936260(127.13.243.2:38235)]}]
02:04:03.108 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=89c23a195e1e46c19d1cff4420b1f6e5, attempt=10, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=321), Traces: [0ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [1ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [1ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [19ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [21ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [22ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [22ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [37ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [40ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [41ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [42ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [43ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [77ms] refreshing cache from master, [77ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [80ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [81ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [83ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [83ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [177ms] refreshing cache from master, [177ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [180ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [182ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [183ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [183ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [317ms] refreshing cache from master, [317ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [320ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=Deferred@116512325(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f87c036bed0@27111bf8 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@457316348 after retry RPC@2080726106) -> (continuation of Deferred@1302608783 after retry RPC@915109346) -> (continuation of Deferred@487003981 after retry RPC@747723797) -> (continuation of Deferred@787260369 after retry RPC@138097645), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4965c8b6 -> wakeup thread Time-limited test -> (continuation of Deferred@457316348 after retry RPC@2080726106) -> (continuation of Deferred@1302608783 after retry RPC@915109346) -> (continuation of Deferred@487003981 after retry RPC@747723797) -> (continuation of Deferred@787260369 after retry RPC@138097645))) after lookup
02:04:03.108 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)] connecting to peer
02:04:03.109 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:04:03.109 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:04:03.109 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985) for tablet 89c23a195e1e46c19d1cff4420b1f6e5: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:04:03.647 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:04:03.647 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421),7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)[L],b123247401d540528f8324235e936260(127.13.243.2:38235)]}]
02:04:03.647 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=89c23a195e1e46c19d1cff4420b1f6e5, attempt=12, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=860), Traces: [0ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [1ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [1ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [19ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [21ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [22ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [22ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [37ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [40ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [41ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [42ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [43ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [77ms] refreshing cache from master, [77ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [80ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [81ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [83ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [83ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [177ms] refreshing cache from master, [177ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [180ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [182ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [183ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [183ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [317ms] refreshing cache from master, [317ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [320ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [321ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [322ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [322ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [857ms] refreshing cache from master, [857ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [860ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=Deferred@116512325(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f87c036bed0@27111bf8 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@457316348 after retry RPC@2080726106) -> (continuation of Deferred@1302608783 after retry RPC@915109346) -> (continuation of Deferred@487003981 after retry RPC@747723797) -> (continuation of Deferred@787260369 after retry RPC@138097645) -> (continuation of Deferred@1682605966 after retry RPC@893351089), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4965c8b6 -> wakeup thread Time-limited test -> (continuation of Deferred@457316348 after retry RPC@2080726106) -> (continuation of Deferred@1302608783 after retry RPC@915109346) -> (continuation of Deferred@487003981 after retry RPC@747723797) -> (continuation of Deferred@787260369 after retry RPC@138097645) -> (continuation of Deferred@1682605966 after retry RPC@893351089))) after lookup
02:04:03.648 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)] connecting to peer
02:04:03.649 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:04:03.650 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:04:03.650 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985) for tablet 89c23a195e1e46c19d1cff4420b1f6e5: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:04:04.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:04.138824 16132 raft_consensus.cc:493] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7f08c11f6dc94dc3aa69febd1051f2e9)
02:04:04.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:04.139021 16132 raft_consensus.cc:515] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:04:04.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:04.140523 16132 leader_election.cc:290] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6422ba2afbf742febb59b258c6ac5101 (127.13.243.1:33421), 7f08c11f6dc94dc3aa69febd1051f2e9 (127.13.243.3:41985)
02:04:04.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:04.143185 15816 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.243.3:41985: connect: Connection refused (error 111)
02:04:04.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:04.144955 15816 leader_election.cc:336] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7f08c11f6dc94dc3aa69febd1051f2e9 (127.13.243.3:41985): Network error: Client connection negotiation failed: client connection to 127.13.243.3:41985: connect: Connection refused (error 111)
02:04:04.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:04.148953 16071 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89c23a195e1e46c19d1cff4420b1f6e5" candidate_uuid: "b123247401d540528f8324235e936260" candidate_term: 8 candidate_status { last_received { term: 7 index: 28 } } ignore_live_leader: false dest_uuid: "6422ba2afbf742febb59b258c6ac5101" is_pre_election: true
02:04:04.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:04.149446 16071 raft_consensus.cc:2468] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b123247401d540528f8324235e936260 in term 6.
02:04:04.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:04.150494 15816 leader_election.cc:304] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6422ba2afbf742febb59b258c6ac5101, b123247401d540528f8324235e936260; no voters: 7f08c11f6dc94dc3aa69febd1051f2e9
02:04:04.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:04.150882 16132 raft_consensus.cc:2804] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [term 7 FOLLOWER]: Leader pre-election won for term 8
02:04:04.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:04.151021 16132 raft_consensus.cc:493] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 7f08c11f6dc94dc3aa69febd1051f2e9)
02:04:04.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:04.151140 16132 raft_consensus.cc:3060] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [term 7 FOLLOWER]: Advancing to term 8
02:04:04.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:04.153367 16132 raft_consensus.cc:515] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:04:04.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:04.153952 16132 leader_election.cc:290] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [CANDIDATE]: Term 8 election: Requested vote from peers 6422ba2afbf742febb59b258c6ac5101 (127.13.243.1:33421), 7f08c11f6dc94dc3aa69febd1051f2e9 (127.13.243.3:41985)
02:04:04.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:04.154532 16071 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89c23a195e1e46c19d1cff4420b1f6e5" candidate_uuid: "b123247401d540528f8324235e936260" candidate_term: 8 candidate_status { last_received { term: 7 index: 28 } } ignore_live_leader: false dest_uuid: "6422ba2afbf742febb59b258c6ac5101"
02:04:04.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:04.154748 16071 raft_consensus.cc:3060] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [term 6 FOLLOWER]: Advancing to term 8
02:04:04.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:04.155541 15816 leader_election.cc:336] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 7f08c11f6dc94dc3aa69febd1051f2e9 (127.13.243.3:41985): Network error: Client connection negotiation failed: client connection to 127.13.243.3:41985: connect: Connection refused (error 111)
02:04:04.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:04.158572 16071 raft_consensus.cc:2468] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b123247401d540528f8324235e936260 in term 8.
02:04:04.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:04.159087 15816 leader_election.cc:304] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6422ba2afbf742febb59b258c6ac5101, b123247401d540528f8324235e936260; no voters: 7f08c11f6dc94dc3aa69febd1051f2e9
02:04:04.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:04.159392 16132 raft_consensus.cc:2804] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [term 8 FOLLOWER]: Leader election won for term 8
02:04:04.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:04.159861 16132 raft_consensus.cc:697] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [term 8 LEADER]: Becoming Leader. State: Replica: b123247401d540528f8324235e936260, State: Running, Role: LEADER
02:04:04.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:04.160409 16132 consensus_queue.cc:237] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 7.28, Last appended by leader: 28, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:04:04.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:04.163961 14468 catalog_manager.cc:5696] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 reported cstate change: term changed from 7 to 8, leader changed from 7f08c11f6dc94dc3aa69febd1051f2e9 (127.13.243.3) to b123247401d540528f8324235e936260 (127.13.243.2). New cstate: current_term: 8 leader_uuid: "b123247401d540528f8324235e936260" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } health_report { overall_health: UNKNOWN } } }
02:04:04.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:04.696069 16071 raft_consensus.cc:1275] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [term 8 FOLLOWER]: Refusing update from remote peer b123247401d540528f8324235e936260: Log matching property violated. Preceding OpId in replica: term: 6 index: 24. Preceding OpId from leader: term: 8 index: 29. (index mismatch)
02:04:04.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:04.697021 16150 consensus_queue.cc:1048] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 24, Time since last communication: 0.000s
02:04:04.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:04.703125 15816 consensus_peers.cc:597] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 -> Peer 7f08c11f6dc94dc3aa69febd1051f2e9 (127.13.243.3:41985): Couldn't send request to peer 7f08c11f6dc94dc3aa69febd1051f2e9. Status: Network error: Client connection negotiation failed: client connection to 127.13.243.3:41985: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
02:04:07.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:07.367880 15816 consensus_peers.cc:597] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 -> Peer 7f08c11f6dc94dc3aa69febd1051f2e9 (127.13.243.3:41985): Couldn't send request to peer 7f08c11f6dc94dc3aa69febd1051f2e9. Status: Network error: Client connection negotiation failed: client connection to 127.13.243.3:41985: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
02:04:07.627 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:04:07.628 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421),7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985),b123247401d540528f8324235e936260(127.13.243.2:38235)[L]]}]
02:04:07.628 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=89c23a195e1e46c19d1cff4420b1f6e5, attempt=14, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=4841), Traces: [0ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [1ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [1ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [19ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [21ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [22ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [22ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [37ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [40ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [41ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [42ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [43ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [77ms] refreshing cache from master, [77ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [80ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [81ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [83ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [83ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [177ms] refreshing cache from master, [177ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [180ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [182ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [183ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [183ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [317ms] refreshing cache from master, [317ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [320ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [321ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [322ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [322ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [857ms] refreshing cache from master, [857ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [860ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [861ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [863ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [863ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [4837ms] refreshing cache from master, [4837ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [4840ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=Deferred@116512325(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f87c036bed0@27111bf8 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@457316348 after retry RPC@2080726106) -> (continuation of Deferred@1302608783 after retry RPC@915109346) -> (continuation of Deferred@487003981 after retry RPC@747723797) -> (continuation of Deferred@787260369 after retry RPC@138097645) -> (continuation of Deferred@1682605966 after retry RPC@893351089) -> (continuation of Deferred@1854767812 after retry RPC@542767448), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4965c8b6 -> wakeup thread Time-limited test -> (continuation of Deferred@457316348 after retry RPC@2080726106) -> (continuation of Deferred@1302608783 after retry RPC@915109346) -> (continuation of Deferred@487003981 after retry RPC@747723797) -> (continuation of Deferred@787260369 after retry RPC@138097645) -> (continuation of Deferred@1682605966 after retry RPC@893351089) -> (continuation of Deferred@1854767812 after retry RPC@542767448))) after lookup
02:04:07.629 [DEBUG - kudu-nio-1] (Connection.java:748) [peer b123247401d540528f8324235e936260(127.13.243.2:38235)] connecting to peer
02:04:07.630 [DEBUG - kudu-nio-5] (Connection.java:757) [peer b123247401d540528f8324235e936260(127.13.243.2:38235)] Successfully connected to peer
02:04:07.648 [DEBUG - kudu-nio-5] (Negotiator.java:919) Authenticated connection [id: 0x8f5260e7, L:/127.0.0.1:46552 - R:/127.13.243.2:38235] using TOKEN/null
02:04:07.674 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.13.243.3:41985
02:04:07.674 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

02:04:07.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:07.674964 14284 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu
02:04:07.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu
02:04:07.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/wal
02:04:07.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/data
02:04:07.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:04:07.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:04:07.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:04:07.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:04:07.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:04:07.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:04:07.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/logs
02:04:07.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/data/info.pb
02:04:07.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:04:07.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:04:07.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:04:07.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:04:07.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:04:07.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:04:07.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:04:07.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:04:07.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:04:07.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:04:07.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.13.243.3:41985
02:04:07.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.13.243.3
02:04:07.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.13.243.62:45883,127.13.243.61:44961,127.13.243.60:35015
02:04:07.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=43625
02:04:07.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.13.243.3
02:04:07.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.13.243.20:34675
02:04:07.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:04:07.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:04:07.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:04:08.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:08.004401 16180 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:04:08.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:08.004719 16180 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:04:08.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:08.004805 16180 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:04:08.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:08.013100 16180 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:04:08.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:08.013276 16180 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.243.3
02:04:08.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:08.023241 16180 tablet_server_runner.cc:78] Tablet server non-default flags:
02:04:08.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:04:08.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.13.243.20:34675
02:04:08.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:04:08.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:04:08.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/data
02:04:08.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/wal
02:04:08.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:04:08.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:04:08.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.13.243.3:41985
02:04:08.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:04:08.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:04:08.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:04:08.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/data/info.pb
02:04:08.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.13.243.3
02:04:08.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=43625
02:04:08.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.13.243.62:45883,127.13.243.61:44961,127.13.243.60:35015
02:04:08.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:04:08.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:04:08.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:04:08.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:04:08.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:04:08.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.13.243.3
02:04:08.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/logs
02:04:08.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:04:08.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:04:08.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:04:08.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:04:08.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:04:08.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision a3f35afc2b6dad6aa403ec58378221544a26edd0
02:04:08.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:04:08.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:40:41 UTC on bdcb31816ec0
02:04:08.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 16849
02:04:08.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
02:04:08.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:08.025363 16180 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:04:08.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:08.027418 16180 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:04:08.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:08.037918 16188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:04:08.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:08.038020 16190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:04:08.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:08.038256 16180 server_base.cc:1061] running on GCE node
02:04:08.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:08.037988 16187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:04:08.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:08.039171 16180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:04:08.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:08.040148 16180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:04:08.041 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:08.041368 16180 hybrid_clock.cc:648] HybridClock initialized: now 1783476248041328 us; error 28 us; skew 500 ppm
02:04:08.041 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:08.041734 16180 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:04:08.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:08.043972 16180 webserver.cc:534] Webserver started at http://127.13.243.3:43625/ using document root <none> and password file <none>
02:04:08.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:08.044732 16180 fs_manager.cc:362] Metadata directory not provided
02:04:08.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:08.044862 16180 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:04:08.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:08.050104 16180 fs_manager.cc:714] Time spent opening directory manager: real 0.003s	user 0.003s	sys 0.000s
02:04:08.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:08.052817 16196 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:04:08.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:08.054457 16180 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
02:04:08.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:08.054656 16180 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/data,/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/wal
02:04:08.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "7f08c11f6dc94dc3aa69febd1051f2e9"
02:04:08.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:03:37 on dist-test-slave-pjt5"
02:04:08.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:08.055570 16180 fs_report.cc:389] FS layout report
02:04:08.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:04:08.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/wal
02:04:08.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/wal
02:04:08.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/data/data
02:04:08.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:04:08.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:04:08.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:04:08.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:04:08.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:04:08.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:04:08.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:04:08.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:04:08.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:04:08.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:04:08.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:04:08.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:08.078049 16180 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:04:08.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:08.079000 16180 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:04:08.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:08.079335 16180 kserver.cc:163] Server-wide thread pool size limit: 3276
02:04:08.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:08.080322 16180 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:04:08.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:08.082523 16203 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
02:04:08.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:08.088524 16180 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
02:04:08.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:08.088658 16180 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.007s	user 0.000s	sys 0.001s
02:04:08.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:08.088800 16180 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
02:04:08.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:08.092046 16180 ts_tablet_manager.cc:616] Registered 1 tablets
02:04:08.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:08.092166 16180 ts_tablet_manager.cc:595] Time spent register tablets: real 0.003s	user 0.002s	sys 0.000s
02:04:08.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:08.092628 16203 tablet_bootstrap.cc:492] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9: Bootstrap starting.
02:04:08.120 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:08.120507 16203 log.cc:826] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9: Log is configured to *not* fsync() on all Append() calls
02:04:08.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:08.140637 16180 rpc_server.cc:307] RPC server started. Bound to: 127.13.243.3:41985
02:04:08.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:08.140693 16310 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.243.3:41985 every 8 connection(s)
02:04:08.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:08.142488 16180 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/data/info.pb
02:04:08.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:08.143262 14284 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu as pid 16180
02:04:08.143 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:04:08.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:08.163852 16203 tablet_bootstrap.cc:492] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=21 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
02:04:08.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:08.164656 16203 tablet_bootstrap.cc:492] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9: Bootstrap complete.
02:04:08.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:08.166450 16203 ts_tablet_manager.cc:1403] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9: Time spent bootstrapping tablet: real 0.074s	user 0.055s	sys 0.016s
02:04:08.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:08.174623 16203 raft_consensus.cc:359] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [term 7 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:04:08.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:08.175717 16203 raft_consensus.cc:740] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f08c11f6dc94dc3aa69febd1051f2e9, State: Initialized, Role: FOLLOWER
02:04:08.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:08.176630 16203 consensus_queue.cc:260] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 7.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:04:08.197 [DEBUG - kudu-nio-5] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 24 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291, tablet=89c23a195e1e46c19d1cff4420b1f6e5, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:04:08.198 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.13.243.2:38235
02:04:08.199 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

02:04:08.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:08.204206 14284 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu pid 15800
02:04:08.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:08.227183 16312 heartbeater.cc:344] Connected to a master server at 127.13.243.62:45883
02:04:08.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:08.227643 16312 heartbeater.cc:461] Registering TS with master...
02:04:08.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:08.228226 16318 heartbeater.cc:344] Connected to a master server at 127.13.243.61:44961
02:04:08.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:08.228374 16318 heartbeater.cc:461] Registering TS with master...
02:04:08.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:08.229055 16312 heartbeater.cc:507] Master 127.13.243.62:45883 requested a full tablet report, sending...
02:04:08.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:08.229152 16318 heartbeater.cc:507] Master 127.13.243.61:44961 requested a full tablet report, sending...
02:04:08.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:08.233305 14397 ts_manager.cc:194] Re-registered known tserver with Master: 7f08c11f6dc94dc3aa69febd1051f2e9 (127.13.243.3:41985)
02:04:08.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:08.233388 14325 ts_manager.cc:194] Re-registered known tserver with Master: 7f08c11f6dc94dc3aa69febd1051f2e9 (127.13.243.3:41985)
02:04:08.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:08.235539 16311 heartbeater.cc:344] Connected to a master server at 127.13.243.60:35015
02:04:08.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:08.235698 16311 heartbeater.cc:461] Registering TS with master...
02:04:08.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:08.236232 16311 heartbeater.cc:507] Master 127.13.243.60:35015 requested a full tablet report, sending...
02:04:08.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:08.237685 14468 ts_manager.cc:194] Re-registered known tserver with Master: 7f08c11f6dc94dc3aa69febd1051f2e9 (127.13.243.3:41985)
02:04:08.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:08.237702 15925 generic_service.cc:196] Checking for leaks (request via RPC)
02:04:08.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:08.239127 14468 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.13.243.3:57689
02:04:08.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:08.240020 16203 ts_tablet_manager.cc:1434] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9: Time spent starting tablet: real 0.073s	user 0.018s	sys 0.029s
02:04:08.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:08.242875 16311 heartbeater.cc:499] Master 127.13.243.60:35015 was elected leader, sending a full tablet report...
02:04:08.341 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:08.341004 14284 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu with pid 15800
02:04:08.362 [DEBUG - kudu-nio-5] (Connection.java:239) [peer b123247401d540528f8324235e936260(127.13.243.2:38235)] handling channelInactive
02:04:08.362 [DEBUG - kudu-nio-5] (Connection.java:709) [peer b123247401d540528f8324235e936260(127.13.243.2:38235)] cleaning up while in state READY due to: connection closed
02:04:08.362 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:04:08.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:08.363106 16326 negotiation.cc:336] Failed RPC negotiation. Trace:
02:04:08.363 [DEBUG - Time-limited test] (Connection.java:748) [peer b123247401d540528f8324235e936260(127.13.243.2:38235)] connecting to peer
02:04:08.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:04:08.341102 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.13.243.2:40799 (local address 127.13.243.3:41985)
02:04:08.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:04:08.349114 (+  8012us) server_negotiation.cc:207] Beginning negotiation
02:04:08.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:04:08.349125 (+    11us) server_negotiation.cc:400] Waiting for connection header
02:04:08.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:04:08.349165 (+    40us) server_negotiation.cc:408] Connection header received
02:04:08.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:04:08.349286 (+   121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
02:04:08.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:04:08.349297 (+    11us) server_negotiation.cc:462] Received NEGOTIATE request from client
02:04:08.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:04:08.349587 (+   290us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
02:04:08.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:04:08.349823 (+   236us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
02:04:08.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:04:08.361708 (+ 11885us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.243.2:40799: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
02:04:08.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":7828,"thread_start_us":7671,"threads_started":1}
02:04:08.364 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235
02:04:08.364 [DEBUG - kudu-nio-6] (Connection.java:709) [peer b123247401d540528f8324235e936260(127.13.243.2:38235)] cleaning up while in state CONNECTING due to: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235
02:04:08.364 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location b123247401d540528f8324235e936260(127.13.243.2:38235) for tablet 89c23a195e1e46c19d1cff4420b1f6e5: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235
02:04:08.387 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:04:08.387 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421),7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985),b123247401d540528f8324235e936260(127.13.243.2:38235)[L]]}]
02:04:08.387 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=89c23a195e1e46c19d1cff4420b1f6e5, attempt=2, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=24), Traces: [0ms] sending RPC to server b123247401d540528f8324235e936260, [1ms] received response from server b123247401d540528f8324235e936260: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [1ms] delaying RPC due to: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [23ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=Deferred@1123149038(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f87c036bed0@2d0d56d1 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5c7218a0 -> wakeup thread Time-limited test)) after lookup
02:04:08.388 [DEBUG - kudu-nio-1] (Connection.java:748) [peer b123247401d540528f8324235e936260(127.13.243.2:38235)] connecting to peer
02:04:08.389 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235
02:04:08.389 [DEBUG - kudu-nio-7] (Connection.java:709) [peer b123247401d540528f8324235e936260(127.13.243.2:38235)] cleaning up while in state CONNECTING due to: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235
02:04:08.389 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location b123247401d540528f8324235e936260(127.13.243.2:38235) for tablet 89c23a195e1e46c19d1cff4420b1f6e5: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235
02:04:08.406 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:04:08.407 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421),7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985),b123247401d540528f8324235e936260(127.13.243.2:38235)[L]]}]
02:04:08.407 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=89c23a195e1e46c19d1cff4420b1f6e5, attempt=4, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=44), Traces: [0ms] sending RPC to server b123247401d540528f8324235e936260, [1ms] received response from server b123247401d540528f8324235e936260: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [1ms] delaying RPC due to: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [23ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [24ms] sending RPC to server b123247401d540528f8324235e936260, [26ms] received response from server b123247401d540528f8324235e936260: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [26ms] delaying RPC due to: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [40ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [43ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=Deferred@1123149038(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f87c036bed0@2d0d56d1 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1356268544 after retry RPC@481079714), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5c7218a0 -> wakeup thread Time-limited test -> (continuation of Deferred@1356268544 after retry RPC@481079714))) after lookup
02:04:08.407 [DEBUG - kudu-nio-1] (Connection.java:748) [peer b123247401d540528f8324235e936260(127.13.243.2:38235)] connecting to peer
02:04:08.408 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235
02:04:08.408 [DEBUG - kudu-nio-0] (Connection.java:709) [peer b123247401d540528f8324235e936260(127.13.243.2:38235)] cleaning up while in state CONNECTING due to: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235
02:04:08.409 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location b123247401d540528f8324235e936260(127.13.243.2:38235) for tablet 89c23a195e1e46c19d1cff4420b1f6e5: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235
02:04:08.427 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:04:08.427 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421),7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985),b123247401d540528f8324235e936260(127.13.243.2:38235)[L]]}]
02:04:08.427 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=89c23a195e1e46c19d1cff4420b1f6e5, attempt=6, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=64), Traces: [0ms] sending RPC to server b123247401d540528f8324235e936260, [1ms] received response from server b123247401d540528f8324235e936260: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [1ms] delaying RPC due to: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [23ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [24ms] sending RPC to server b123247401d540528f8324235e936260, [26ms] received response from server b123247401d540528f8324235e936260: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [26ms] delaying RPC due to: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [40ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [43ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [44ms] sending RPC to server b123247401d540528f8324235e936260, [46ms] received response from server b123247401d540528f8324235e936260: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [46ms] delaying RPC due to: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [61ms] refreshing cache from master, [61ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [64ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=Deferred@1123149038(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f87c036bed0@2d0d56d1 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1356268544 after retry RPC@481079714) -> (continuation of Deferred@1668690745 after retry RPC@439267156), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5c7218a0 -> wakeup thread Time-limited test -> (continuation of Deferred@1356268544 after retry RPC@481079714) -> (continuation of Deferred@1668690745 after retry RPC@439267156))) after lookup
02:04:08.428 [DEBUG - kudu-nio-1] (Connection.java:748) [peer b123247401d540528f8324235e936260(127.13.243.2:38235)] connecting to peer
02:04:08.428 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235
02:04:08.429 [DEBUG - kudu-nio-1] (Connection.java:709) [peer b123247401d540528f8324235e936260(127.13.243.2:38235)] cleaning up while in state CONNECTING due to: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235
02:04:08.429 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location b123247401d540528f8324235e936260(127.13.243.2:38235) for tablet 89c23a195e1e46c19d1cff4420b1f6e5: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235
02:04:08.467 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:04:08.467 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421),7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985),b123247401d540528f8324235e936260(127.13.243.2:38235)[L]]}]
02:04:08.467 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=89c23a195e1e46c19d1cff4420b1f6e5, attempt=8, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=104), Traces: [0ms] sending RPC to server b123247401d540528f8324235e936260, [1ms] received response from server b123247401d540528f8324235e936260: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [1ms] delaying RPC due to: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [23ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [24ms] sending RPC to server b123247401d540528f8324235e936260, [26ms] received response from server b123247401d540528f8324235e936260: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [26ms] delaying RPC due to: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [40ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [43ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [44ms] sending RPC to server b123247401d540528f8324235e936260, [46ms] received response from server b123247401d540528f8324235e936260: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [46ms] delaying RPC due to: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [61ms] refreshing cache from master, [61ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [64ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [64ms] sending RPC to server b123247401d540528f8324235e936260, [66ms] received response from server b123247401d540528f8324235e936260: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [66ms] delaying RPC due to: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [101ms] refreshing cache from master, [101ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [103ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=Deferred@1123149038(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f87c036bed0@2d0d56d1 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1356268544 after retry RPC@481079714) -> (continuation of Deferred@1668690745 after retry RPC@439267156) -> (continuation of Deferred@1914370663 after retry RPC@1538292661), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5c7218a0 -> wakeup thread Time-limited test -> (continuation of Deferred@1356268544 after retry RPC@481079714) -> (continuation of Deferred@1668690745 after retry RPC@439267156) -> (continuation of Deferred@1914370663 after retry RPC@1538292661))) after lookup
02:04:08.468 [DEBUG - kudu-nio-1] (Connection.java:748) [peer b123247401d540528f8324235e936260(127.13.243.2:38235)] connecting to peer
02:04:08.469 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235
02:04:08.469 [DEBUG - kudu-nio-2] (Connection.java:709) [peer b123247401d540528f8324235e936260(127.13.243.2:38235)] cleaning up while in state CONNECTING due to: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235
02:04:08.469 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location b123247401d540528f8324235e936260(127.13.243.2:38235) for tablet 89c23a195e1e46c19d1cff4420b1f6e5: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235
02:04:08.867 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:04:08.867 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421),7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985),b123247401d540528f8324235e936260(127.13.243.2:38235)[L]]}]
02:04:08.867 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=89c23a195e1e46c19d1cff4420b1f6e5, attempt=10, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=504), Traces: [0ms] sending RPC to server b123247401d540528f8324235e936260, [1ms] received response from server b123247401d540528f8324235e936260: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [1ms] delaying RPC due to: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [23ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [24ms] sending RPC to server b123247401d540528f8324235e936260, [26ms] received response from server b123247401d540528f8324235e936260: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [26ms] delaying RPC due to: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [40ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [43ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [44ms] sending RPC to server b123247401d540528f8324235e936260, [46ms] received response from server b123247401d540528f8324235e936260: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [46ms] delaying RPC due to: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [61ms] refreshing cache from master, [61ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [64ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [64ms] sending RPC to server b123247401d540528f8324235e936260, [66ms] received response from server b123247401d540528f8324235e936260: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [66ms] delaying RPC due to: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [101ms] refreshing cache from master, [101ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [103ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [104ms] sending RPC to server b123247401d540528f8324235e936260, [106ms] received response from server b123247401d540528f8324235e936260: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [106ms] delaying RPC due to: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [501ms] refreshing cache from master, [501ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [503ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=Deferred@1123149038(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f87c036bed0@2d0d56d1 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1356268544 after retry RPC@481079714) -> (continuation of Deferred@1668690745 after retry RPC@439267156) -> (continuation of Deferred@1914370663 after retry RPC@1538292661) -> (continuation of Deferred@1271716332 after retry RPC@326729459), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5c7218a0 -> wakeup thread Time-limited test -> (continuation of Deferred@1356268544 after retry RPC@481079714) -> (continuation of Deferred@1668690745 after retry RPC@439267156) -> (continuation of Deferred@1914370663 after retry RPC@1538292661) -> (continuation of Deferred@1271716332 after retry RPC@326729459))) after lookup
02:04:08.868 [DEBUG - kudu-nio-1] (Connection.java:748) [peer b123247401d540528f8324235e936260(127.13.243.2:38235)] connecting to peer
02:04:08.869 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235
02:04:08.869 [DEBUG - kudu-nio-3] (Connection.java:709) [peer b123247401d540528f8324235e936260(127.13.243.2:38235)] cleaning up while in state CONNECTING due to: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235
02:04:08.869 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location b123247401d540528f8324235e936260(127.13.243.2:38235) for tablet 89c23a195e1e46c19d1cff4420b1f6e5: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235
02:04:09.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:09.465736 16327 raft_consensus.cc:493] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
02:04:09.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:09.466064 16327 raft_consensus.cc:515] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:04:09.468 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:09.467869 16327 leader_election.cc:290] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6422ba2afbf742febb59b258c6ac5101 (127.13.243.1:33421), b123247401d540528f8324235e936260 (127.13.243.2:38235)
02:04:09.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:09.469494 16199 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.243.2:38235: connect: Connection refused (error 111)
02:04:09.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:09.471403 16199 leader_election.cc:336] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer b123247401d540528f8324235e936260 (127.13.243.2:38235): Network error: Client connection negotiation failed: client connection to 127.13.243.2:38235: connect: Connection refused (error 111)
02:04:09.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:09.475057 16071 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89c23a195e1e46c19d1cff4420b1f6e5" candidate_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" candidate_term: 8 candidate_status { last_received { term: 7 index: 28 } } ignore_live_leader: false dest_uuid: "6422ba2afbf742febb59b258c6ac5101" is_pre_election: true
02:04:09.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:09.475858 16197 leader_election.cc:304] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f08c11f6dc94dc3aa69febd1051f2e9; no voters: 6422ba2afbf742febb59b258c6ac5101, b123247401d540528f8324235e936260
02:04:09.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:09.476212 16327 raft_consensus.cc:2749] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
02:04:09.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:09.679055 16332 raft_consensus.cc:493] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b123247401d540528f8324235e936260)
02:04:09.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:09.679270 16332 raft_consensus.cc:515] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:04:09.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:09.680850 16332 leader_election.cc:290] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b123247401d540528f8324235e936260 (127.13.243.2:38235), 7f08c11f6dc94dc3aa69febd1051f2e9 (127.13.243.3:41985)
02:04:09.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:09.682353 16005 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.243.2:38235: connect: Connection refused (error 111)
02:04:09.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:09.684180 16005 leader_election.cc:336] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer b123247401d540528f8324235e936260 (127.13.243.2:38235): Network error: Client connection negotiation failed: client connection to 127.13.243.2:38235: connect: Connection refused (error 111)
02:04:09.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:09.689002 16265 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89c23a195e1e46c19d1cff4420b1f6e5" candidate_uuid: "6422ba2afbf742febb59b258c6ac5101" candidate_term: 9 candidate_status { last_received { term: 8 index: 32 } } ignore_live_leader: false dest_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" is_pre_election: true
02:04:09.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:09.689363 16265 raft_consensus.cc:2468] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6422ba2afbf742febb59b258c6ac5101 in term 7.
02:04:09.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:09.690224 16003 leader_election.cc:304] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6422ba2afbf742febb59b258c6ac5101, 7f08c11f6dc94dc3aa69febd1051f2e9; no voters: b123247401d540528f8324235e936260
02:04:09.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:09.690600 16332 raft_consensus.cc:2804] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [term 8 FOLLOWER]: Leader pre-election won for term 9
02:04:09.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:09.690740 16332 raft_consensus.cc:493] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [term 8 FOLLOWER]: Starting leader election (detected failure of leader b123247401d540528f8324235e936260)
02:04:09.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:09.690843 16332 raft_consensus.cc:3060] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [term 8 FOLLOWER]: Advancing to term 9
02:04:09.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:09.692893 16332 raft_consensus.cc:515] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:04:09.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:09.693507 16332 leader_election.cc:290] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [CANDIDATE]: Term 9 election: Requested vote from peers b123247401d540528f8324235e936260 (127.13.243.2:38235), 7f08c11f6dc94dc3aa69febd1051f2e9 (127.13.243.3:41985)
02:04:09.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:09.694183 16265 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89c23a195e1e46c19d1cff4420b1f6e5" candidate_uuid: "6422ba2afbf742febb59b258c6ac5101" candidate_term: 9 candidate_status { last_received { term: 8 index: 32 } } ignore_live_leader: false dest_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9"
02:04:09.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:09.694427 16265 raft_consensus.cc:3060] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [term 7 FOLLOWER]: Advancing to term 9
02:04:09.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:09.694983 16005 leader_election.cc:336] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer b123247401d540528f8324235e936260 (127.13.243.2:38235): Network error: Client connection negotiation failed: client connection to 127.13.243.2:38235: connect: Connection refused (error 111)
02:04:09.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:09.698179 16265 raft_consensus.cc:2468] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6422ba2afbf742febb59b258c6ac5101 in term 9.
02:04:09.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:09.698639 16003 leader_election.cc:304] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6422ba2afbf742febb59b258c6ac5101, 7f08c11f6dc94dc3aa69febd1051f2e9; no voters: b123247401d540528f8324235e936260
02:04:09.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:09.698963 16332 raft_consensus.cc:2804] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [term 9 FOLLOWER]: Leader election won for term 9
02:04:09.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:09.699391 16332 raft_consensus.cc:697] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [term 9 LEADER]: Becoming Leader. State: Replica: 6422ba2afbf742febb59b258c6ac5101, State: Running, Role: LEADER
02:04:09.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:09.700124 16332 consensus_queue.cc:237] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 8.32, Last appended by leader: 32, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:04:09.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:09.703904 14468 catalog_manager.cc:5696] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 reported cstate change: term changed from 8 to 9, leader changed from b123247401d540528f8324235e936260 (127.13.243.2) to 6422ba2afbf742febb59b258c6ac5101 (127.13.243.1). New cstate: current_term: 9 leader_uuid: "6422ba2afbf742febb59b258c6ac5101" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } health_report { overall_health: UNKNOWN } } }
02:04:09.988 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:04:09.988 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)[L],7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985),b123247401d540528f8324235e936260(127.13.243.2:38235)]}]
02:04:09.989 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=89c23a195e1e46c19d1cff4420b1f6e5, attempt=12, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=1626), Traces: [0ms] sending RPC to server b123247401d540528f8324235e936260, [1ms] received response from server b123247401d540528f8324235e936260: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [1ms] delaying RPC due to: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [23ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [24ms] sending RPC to server b123247401d540528f8324235e936260, [26ms] received response from server b123247401d540528f8324235e936260: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [26ms] delaying RPC due to: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [40ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [43ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [44ms] sending RPC to server b123247401d540528f8324235e936260, [46ms] received response from server b123247401d540528f8324235e936260: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [46ms] delaying RPC due to: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [61ms] refreshing cache from master, [61ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [64ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [64ms] sending RPC to server b123247401d540528f8324235e936260, [66ms] received response from server b123247401d540528f8324235e936260: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [66ms] delaying RPC due to: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [101ms] refreshing cache from master, [101ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [103ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [104ms] sending RPC to server b123247401d540528f8324235e936260, [106ms] received response from server b123247401d540528f8324235e936260: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [106ms] delaying RPC due to: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [501ms] refreshing cache from master, [501ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [503ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [504ms] sending RPC to server b123247401d540528f8324235e936260, [506ms] received response from server b123247401d540528f8324235e936260: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [506ms] delaying RPC due to: Network error: Failed to connect to peer b123247401d540528f8324235e936260(127.13.243.2:38235): Connection refused: /127.13.243.2:38235, [1621ms] refreshing cache from master, [1621ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [1624ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=Deferred@1123149038(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f87c036bed0@2d0d56d1 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1356268544 after retry RPC@481079714) -> (continuation of Deferred@1668690745 after retry RPC@439267156) -> (continuation of Deferred@1914370663 after retry RPC@1538292661) -> (continuation of Deferred@1271716332 after retry RPC@326729459) -> (continuation of Deferred@1951133379 after retry RPC@740132959), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5c7218a0 -> wakeup thread Time-limited test -> (continuation of Deferred@1356268544 after retry RPC@481079714) -> (continuation of Deferred@1668690745 after retry RPC@439267156) -> (continuation of Deferred@1914370663 after retry RPC@1538292661) -> (continuation of Deferred@1271716332 after retry RPC@326729459) -> (continuation of Deferred@1951133379 after retry RPC@740132959))) after lookup
02:04:09.989 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)] connecting to peer
02:04:09.991 [DEBUG - kudu-nio-4] (Connection.java:757) [peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)] Successfully connected to peer
02:04:10.006 [DEBUG - kudu-nio-4] (Negotiator.java:919) Authenticated connection [id: 0x9c6d4927, L:/127.0.0.1:45416 - R:/127.13.243.1:33421] using TOKEN/null
02:04:10.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:10.012552 16265 raft_consensus.cc:1275] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [term 9 FOLLOWER]: Refusing update from remote peer 6422ba2afbf742febb59b258c6ac5101: Log matching property violated. Preceding OpId in replica: term: 7 index: 28. Preceding OpId from leader: term: 9 index: 34. (index mismatch)
02:04:10.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:10.012908 16005 consensus_peers.cc:597] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 -> Peer b123247401d540528f8324235e936260 (127.13.243.2:38235): Couldn't send request to peer b123247401d540528f8324235e936260. Status: Network error: Client connection negotiation failed: client connection to 127.13.243.2:38235: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
02:04:10.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:10.013557 16332 consensus_queue.cc:1048] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 28, Time since last communication: 0.000s
02:04:10.048 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.13.243.2:38235
02:04:10.049 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

02:04:10.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:10.049775 14284 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu
02:04:10.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu
02:04:10.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-1/wal
02:04:10.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-1/data
02:04:10.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:04:10.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:04:10.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:04:10.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:04:10.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:04:10.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:04:10.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-1/logs
02:04:10.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-1/data/info.pb
02:04:10.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:04:10.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:04:10.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:04:10.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:04:10.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:04:10.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:04:10.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:04:10.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:04:10.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:04:10.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:04:10.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.13.243.2:38235
02:04:10.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.13.243.2
02:04:10.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.13.243.62:45883,127.13.243.61:44961,127.13.243.60:35015
02:04:10.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42383
02:04:10.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.13.243.2
02:04:10.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.13.243.20:34675
02:04:10.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:04:10.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:04:10.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:04:10.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:10.399971 16358 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:04:10.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:10.400374 16358 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:04:10.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:10.400496 16358 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:04:10.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:10.409113 16358 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:04:10.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:10.409361 16358 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.243.2
02:04:10.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:10.419838 16358 tablet_server_runner.cc:78] Tablet server non-default flags:
02:04:10.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:04:10.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.13.243.20:34675
02:04:10.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:04:10.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:04:10.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-1/data
02:04:10.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-1/wal
02:04:10.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:04:10.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:04:10.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.13.243.2:38235
02:04:10.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:04:10.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:04:10.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:04:10.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-1/data/info.pb
02:04:10.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.13.243.2
02:04:10.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42383
02:04:10.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.13.243.62:45883,127.13.243.61:44961,127.13.243.60:35015
02:04:10.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:04:10.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:04:10.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:04:10.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:04:10.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:04:10.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.13.243.2
02:04:10.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-1/logs
02:04:10.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:04:10.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:04:10.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:04:10.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:04:10.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:04:10.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision a3f35afc2b6dad6aa403ec58378221544a26edd0
02:04:10.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:04:10.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:40:41 UTC on bdcb31816ec0
02:04:10.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 16849
02:04:10.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
02:04:10.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:10.422240 16358 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:04:10.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:10.424513 16358 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:04:10.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:10.435758 16364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:04:10.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:10.435721 16363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:04:10.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:10.436592 16366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:04:10.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:10.437438 16358 server_base.cc:1061] running on GCE node
02:04:10.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:10.438011 16358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:04:10.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:10.439086 16358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:04:10.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:10.440301 16358 hybrid_clock.cc:648] HybridClock initialized: now 1783476250440233 us; error 62 us; skew 500 ppm
02:04:10.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:10.440593 16358 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:04:10.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:10.443111 16358 webserver.cc:534] Webserver started at http://127.13.243.2:42383/ using document root <none> and password file <none>
02:04:10.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:10.443898 16358 fs_manager.cc:362] Metadata directory not provided
02:04:10.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:10.444000 16358 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:04:10.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:10.449437 16358 fs_manager.cc:714] Time spent opening directory manager: real 0.004s	user 0.005s	sys 0.000s
02:04:10.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:10.452710 16372 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:04:10.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:10.454454 16358 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
02:04:10.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:10.454634 16358 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-1/data,/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-1/wal
02:04:10.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "b123247401d540528f8324235e936260"
02:04:10.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:03:37 on dist-test-slave-pjt5"
02:04:10.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:10.455386 16358 fs_report.cc:389] FS layout report
02:04:10.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:04:10.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-1/wal
02:04:10.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-1/wal
02:04:10.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-1/data/data
02:04:10.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:04:10.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:04:10.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:04:10.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:04:10.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:04:10.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:04:10.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:04:10.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:04:10.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:04:10.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:04:10.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:04:10.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:10.474865 16358 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:04:10.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:10.475847 16358 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:04:10.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:10.476091 16358 kserver.cc:163] Server-wide thread pool size limit: 3276
02:04:10.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:10.477077 16358 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:04:10.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:10.479524 16379 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
02:04:10.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:10.486090 16358 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
02:04:10.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:10.486274 16358 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.007s	user 0.001s	sys 0.000s
02:04:10.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:10.486414 16358 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
02:04:10.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:10.489778 16358 ts_tablet_manager.cc:616] Registered 1 tablets
02:04:10.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:10.489876 16358 ts_tablet_manager.cc:595] Time spent register tablets: real 0.003s	user 0.000s	sys 0.000s
02:04:10.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:10.490558 16379 tablet_bootstrap.cc:492] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260: Bootstrap starting.
02:04:10.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:10.519626 16379 log.cc:826] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260: Log is configured to *not* fsync() on all Append() calls
02:04:10.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:10.566843 16358 rpc_server.cc:307] RPC server started. Bound to: 127.13.243.2:38235
02:04:10.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:10.566910 16486 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.243.2:38235 every 8 connection(s)
02:04:10.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:10.568969 16358 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-1/data/info.pb
02:04:10.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:10.569494 16379 tablet_bootstrap.cc:492] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=24 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
02:04:10.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:10.569478 14284 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu as pid 16358
02:04:10.570 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:04:10.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:10.570153 16379 tablet_bootstrap.cc:492] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260: Bootstrap complete.
02:04:10.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:10.571381 16379 ts_tablet_manager.cc:1403] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260: Time spent bootstrapping tablet: real 0.081s	user 0.051s	sys 0.025s
02:04:10.596 [DEBUG - kudu-nio-4] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 27 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291, tablet=89c23a195e1e46c19d1cff4420b1f6e5, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:04:10.598 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.13.243.1:33421
02:04:10.598 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

02:04:10.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:10.598924 14284 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu pid 15986
02:04:10.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:10.589466 16379 raft_consensus.cc:359] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [term 8 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:04:10.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:10.618781 16379 raft_consensus.cc:740] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: b123247401d540528f8324235e936260, State: Initialized, Role: FOLLOWER
02:04:10.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:10.620008 16379 consensus_queue.cc:260] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 8.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:04:10.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:10.633728 16489 heartbeater.cc:344] Connected to a master server at 127.13.243.61:44961
02:04:10.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:10.634585 16489 heartbeater.cc:461] Registering TS with master...
02:04:10.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:10.634912 16379 ts_tablet_manager.cc:1434] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260: Time spent starting tablet: real 0.063s	user 0.010s	sys 0.004s
02:04:10.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:10.635669 16488 heartbeater.cc:344] Connected to a master server at 127.13.243.62:45883
02:04:10.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:10.635854 16488 heartbeater.cc:461] Registering TS with master...
02:04:10.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:10.636425 16488 heartbeater.cc:507] Master 127.13.243.62:45883 requested a full tablet report, sending...
02:04:10.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:10.635852 16489 heartbeater.cc:507] Master 127.13.243.61:44961 requested a full tablet report, sending...
02:04:10.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:10.639038 16487 heartbeater.cc:344] Connected to a master server at 127.13.243.60:35015
02:04:10.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:10.639185 16487 heartbeater.cc:461] Registering TS with master...
02:04:10.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:10.639416 14325 ts_manager.cc:194] Re-registered known tserver with Master: b123247401d540528f8324235e936260 (127.13.243.2:38235)
02:04:10.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:10.639674 16487 heartbeater.cc:507] Master 127.13.243.60:35015 requested a full tablet report, sending...
02:04:10.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:10.640916 14397 ts_manager.cc:194] Re-registered known tserver with Master: b123247401d540528f8324235e936260 (127.13.243.2:38235)
02:04:10.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:10.640966 14468 ts_manager.cc:194] Re-registered known tserver with Master: b123247401d540528f8324235e936260 (127.13.243.2:38235)
02:04:10.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:10.642554 14468 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.13.243.2:47125
02:04:10.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:10.646031 16487 heartbeater.cc:499] Master 127.13.243.60:35015 was elected leader, sending a full tablet report...
02:04:10.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:10.647555 16112 generic_service.cc:196] Checking for leaks (request via RPC)
02:04:10.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:10.755318 14284 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu with pid 15986
02:04:10.773 [DEBUG - kudu-nio-4] (Connection.java:239) [peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)] handling channelInactive
02:04:10.774 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)] cleaning up while in state READY due to: connection closed
02:04:10.774 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:04:10.775 [DEBUG - Time-limited test] (Connection.java:748) [peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)] connecting to peer
02:04:10.776 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421
02:04:10.776 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)] cleaning up while in state CONNECTING due to: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421
02:04:10.776 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421) for tablet 89c23a195e1e46c19d1cff4420b1f6e5: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421
02:04:10.786 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:04:10.787 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)[L],7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985),b123247401d540528f8324235e936260(127.13.243.2:38235)]}]
02:04:10.787 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=89c23a195e1e46c19d1cff4420b1f6e5, attempt=2, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=12), Traces: [0ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [2ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [2ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [12ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=Deferred@1276669417(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f87c036bed0@c074b45 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3127c8de -> wakeup thread Time-limited test)) after lookup
02:04:10.787 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)] connecting to peer
02:04:10.788 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421
02:04:10.788 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)] cleaning up while in state CONNECTING due to: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421
02:04:10.789 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421) for tablet 89c23a195e1e46c19d1cff4420b1f6e5: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421
02:04:10.807 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:04:10.807 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)[L],7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985),b123247401d540528f8324235e936260(127.13.243.2:38235)]}]
02:04:10.808 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=89c23a195e1e46c19d1cff4420b1f6e5, attempt=4, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=33), Traces: [0ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [2ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [2ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [12ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [13ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [14ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [15ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [30ms] refreshing cache from master, [30ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [33ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=Deferred@1276669417(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f87c036bed0@c074b45 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@452574864 after retry RPC@353887273), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3127c8de -> wakeup thread Time-limited test -> (continuation of Deferred@452574864 after retry RPC@353887273))) after lookup
02:04:10.808 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)] connecting to peer
02:04:10.810 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421
02:04:10.810 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)] cleaning up while in state CONNECTING due to: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421
02:04:10.810 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421) for tablet 89c23a195e1e46c19d1cff4420b1f6e5: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421
02:04:10.827 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:04:10.827 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)[L],7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985),b123247401d540528f8324235e936260(127.13.243.2:38235)]}]
02:04:10.827 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=89c23a195e1e46c19d1cff4420b1f6e5, attempt=6, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=52), Traces: [0ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [2ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [2ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [12ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [13ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [14ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [15ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [30ms] refreshing cache from master, [30ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [33ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [34ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [36ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [36ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [52ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=Deferred@1276669417(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f87c036bed0@c074b45 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@452574864 after retry RPC@353887273) -> (continuation of Deferred@626117286 after retry RPC@1717126826), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3127c8de -> wakeup thread Time-limited test -> (continuation of Deferred@452574864 after retry RPC@353887273) -> (continuation of Deferred@626117286 after retry RPC@1717126826))) after lookup
02:04:10.828 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)] connecting to peer
02:04:10.829 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421
02:04:10.829 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)] cleaning up while in state CONNECTING due to: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421
02:04:10.829 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421) for tablet 89c23a195e1e46c19d1cff4420b1f6e5: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421
02:04:10.967 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:04:10.968 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)[L],7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985),b123247401d540528f8324235e936260(127.13.243.2:38235)]}]
02:04:10.968 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=89c23a195e1e46c19d1cff4420b1f6e5, attempt=8, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=193), Traces: [0ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [2ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [2ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [12ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [13ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [14ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [15ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [30ms] refreshing cache from master, [30ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [33ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [34ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [36ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [36ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [52ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [54ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [55ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [55ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [190ms] refreshing cache from master, [190ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [193ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=Deferred@1276669417(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f87c036bed0@c074b45 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@452574864 after retry RPC@353887273) -> (continuation of Deferred@626117286 after retry RPC@1717126826) -> (continuation of Deferred@1265631229 after retry RPC@1731169324), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3127c8de -> wakeup thread Time-limited test -> (continuation of Deferred@452574864 after retry RPC@353887273) -> (continuation of Deferred@626117286 after retry RPC@1717126826) -> (continuation of Deferred@1265631229 after retry RPC@1731169324))) after lookup
02:04:10.968 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)] connecting to peer
02:04:10.969 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421
02:04:10.970 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)] cleaning up while in state CONNECTING due to: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421
02:04:10.970 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421) for tablet 89c23a195e1e46c19d1cff4420b1f6e5: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421
02:04:11.487 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:04:11.488 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)[L],7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985),b123247401d540528f8324235e936260(127.13.243.2:38235)]}]
02:04:11.488 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=89c23a195e1e46c19d1cff4420b1f6e5, attempt=10, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=713), Traces: [0ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [2ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [2ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [12ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [13ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [14ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [15ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [30ms] refreshing cache from master, [30ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [33ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [34ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [36ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [36ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [52ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [54ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [55ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [55ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [190ms] refreshing cache from master, [190ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [193ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [194ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [196ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [196ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [710ms] refreshing cache from master, [710ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [713ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=Deferred@1276669417(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f87c036bed0@c074b45 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@452574864 after retry RPC@353887273) -> (continuation of Deferred@626117286 after retry RPC@1717126826) -> (continuation of Deferred@1265631229 after retry RPC@1731169324) -> (continuation of Deferred@1794392018 after retry RPC@351117501), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3127c8de -> wakeup thread Time-limited test -> (continuation of Deferred@452574864 after retry RPC@353887273) -> (continuation of Deferred@626117286 after retry RPC@1717126826) -> (continuation of Deferred@1265631229 after retry RPC@1731169324) -> (continuation of Deferred@1794392018 after retry RPC@351117501))) after lookup
02:04:11.489 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)] connecting to peer
02:04:11.490 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421
02:04:11.490 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)] cleaning up while in state CONNECTING due to: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421
02:04:11.490 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421) for tablet 89c23a195e1e46c19d1cff4420b1f6e5: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421
02:04:11.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:11.961616 16502 raft_consensus.cc:493] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 6422ba2afbf742febb59b258c6ac5101)
02:04:11.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:11.961874 16502 raft_consensus.cc:515] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:04:11.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:11.962680 16502 leader_election.cc:290] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6422ba2afbf742febb59b258c6ac5101 (127.13.243.1:33421), b123247401d540528f8324235e936260 (127.13.243.2:38235)
02:04:11.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:11.965505 16197 leader_election.cc:336] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 6422ba2afbf742febb59b258c6ac5101 (127.13.243.1:33421): Network error: Client connection negotiation failed: client connection to 127.13.243.1:33421: connect: Connection refused (error 111)
02:04:11.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:11.970712 16442 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89c23a195e1e46c19d1cff4420b1f6e5" candidate_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "b123247401d540528f8324235e936260" is_pre_election: true
02:04:11.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:11.971240 16442 raft_consensus.cc:2468] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f08c11f6dc94dc3aa69febd1051f2e9 in term 8.
02:04:11.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:11.972319 16199 leader_election.cc:304] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7f08c11f6dc94dc3aa69febd1051f2e9, b123247401d540528f8324235e936260; no voters: 6422ba2afbf742febb59b258c6ac5101
02:04:11.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:11.972666 16502 raft_consensus.cc:2804] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [term 9 FOLLOWER]: Leader pre-election won for term 10
02:04:11.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:11.972812 16502 raft_consensus.cc:493] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 6422ba2afbf742febb59b258c6ac5101)
02:04:11.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:11.972918 16502 raft_consensus.cc:3060] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [term 9 FOLLOWER]: Advancing to term 10
02:04:11.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:11.975308 16502 raft_consensus.cc:515] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:04:11.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:11.975965 16502 leader_election.cc:290] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [CANDIDATE]: Term 10 election: Requested vote from peers 6422ba2afbf742febb59b258c6ac5101 (127.13.243.1:33421), b123247401d540528f8324235e936260 (127.13.243.2:38235)
02:04:11.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:11.976544 16442 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89c23a195e1e46c19d1cff4420b1f6e5" candidate_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "b123247401d540528f8324235e936260"
02:04:11.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:11.976771 16442 raft_consensus.cc:3060] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [term 8 FOLLOWER]: Advancing to term 10
02:04:11.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:11.977520 16197 leader_election.cc:336] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 6422ba2afbf742febb59b258c6ac5101 (127.13.243.1:33421): Network error: Client connection negotiation failed: client connection to 127.13.243.1:33421: connect: Connection refused (error 111)
02:04:11.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:11.980778 16442 raft_consensus.cc:2468] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f08c11f6dc94dc3aa69febd1051f2e9 in term 10.
02:04:11.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:11.981314 16199 leader_election.cc:304] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7f08c11f6dc94dc3aa69febd1051f2e9, b123247401d540528f8324235e936260; no voters: 6422ba2afbf742febb59b258c6ac5101
02:04:11.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:11.981676 16502 raft_consensus.cc:2804] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [term 10 FOLLOWER]: Leader election won for term 10
02:04:11.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:11.982152 16502 raft_consensus.cc:697] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [term 10 LEADER]: Becoming Leader. State: Replica: 7f08c11f6dc94dc3aa69febd1051f2e9, State: Running, Role: LEADER
02:04:11.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:11.982748 16502 consensus_queue.cc:237] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 9.36, Last appended by leader: 36, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:04:11.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:11.986850 14468 catalog_manager.cc:5696] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 reported cstate change: term changed from 9 to 10, leader changed from 6422ba2afbf742febb59b258c6ac5101 (127.13.243.1) to 7f08c11f6dc94dc3aa69febd1051f2e9 (127.13.243.3). New cstate: current_term: 10 leader_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } health_report { overall_health: HEALTHY } } }
02:04:12.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:12.513149 16442 raft_consensus.cc:1275] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [term 10 FOLLOWER]: Refusing update from remote peer 7f08c11f6dc94dc3aa69febd1051f2e9: Log matching property violated. Preceding OpId in replica: term: 8 index: 32. Preceding OpId from leader: term: 10 index: 37. (index mismatch)
02:04:12.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:12.514051 16520 consensus_queue.cc:1048] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 32, Time since last communication: 0.000s
02:04:12.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:12.530611 16197 consensus_peers.cc:597] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 -> Peer 6422ba2afbf742febb59b258c6ac5101 (127.13.243.1:33421): Couldn't send request to peer 6422ba2afbf742febb59b258c6ac5101. Status: Network error: Client connection negotiation failed: client connection to 127.13.243.1:33421: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
02:04:13.527 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:04:13.527 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421),7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)[L],b123247401d540528f8324235e936260(127.13.243.2:38235)]}]
02:04:13.531 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=89c23a195e1e46c19d1cff4420b1f6e5, attempt=12, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=2756), Traces: [0ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [2ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [2ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [12ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [13ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [14ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [15ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [30ms] refreshing cache from master, [30ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [33ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [34ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [36ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [36ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [52ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [54ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [55ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [55ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [190ms] refreshing cache from master, [190ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [193ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [194ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [196ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [196ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [710ms] refreshing cache from master, [710ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [713ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [714ms] sending RPC to server 6422ba2afbf742febb59b258c6ac5101, [716ms] received response from server 6422ba2afbf742febb59b258c6ac5101: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [716ms] delaying RPC due to: Network error: Failed to connect to peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421): Connection refused: /127.13.243.1:33421, [2750ms] refreshing cache from master, [2750ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [2752ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=Deferred@1276669417(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f87c036bed0@c074b45 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@452574864 after retry RPC@353887273) -> (continuation of Deferred@626117286 after retry RPC@1717126826) -> (continuation of Deferred@1265631229 after retry RPC@1731169324) -> (continuation of Deferred@1794392018 after retry RPC@351117501) -> (continuation of Deferred@1767441856 after retry RPC@273197086), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3127c8de -> wakeup thread Time-limited test -> (continuation of Deferred@452574864 after retry RPC@353887273) -> (continuation of Deferred@626117286 after retry RPC@1717126826) -> (continuation of Deferred@1265631229 after retry RPC@1731169324) -> (continuation of Deferred@1794392018 after retry RPC@351117501) -> (continuation of Deferred@1767441856 after retry RPC@273197086))) after lookup
02:04:13.531 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)] connecting to peer
02:04:13.535 [DEBUG - kudu-nio-3] (Connection.java:757) [peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)] Successfully connected to peer
02:04:13.551 [DEBUG - kudu-nio-3] (Negotiator.java:919) Authenticated connection [id: 0xfaca8a9e, L:/127.0.0.1:60084 - R:/127.13.243.3:41985] using TOKEN/null
02:04:13.572 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.13.243.1:33421
02:04:13.573 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

02:04:13.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:13.573871 14284 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu
02:04:13.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu
02:04:13.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-0/wal
02:04:13.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-0/data
02:04:13.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:04:13.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:04:13.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:04:13.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:04:13.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:04:13.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:04:13.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-0/logs
02:04:13.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-0/data/info.pb
02:04:13.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:04:13.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:04:13.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:04:13.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:04:13.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:04:13.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:04:13.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:04:13.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:04:13.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:04:13.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:04:13.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.13.243.1:33421
02:04:13.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.13.243.1
02:04:13.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.13.243.62:45883,127.13.243.61:44961,127.13.243.60:35015
02:04:13.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42315
02:04:13.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.13.243.1
02:04:13.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.13.243.20:34675
02:04:13.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:04:13.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:04:13.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:04:13.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:13.902091 16542 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:04:13.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:13.902462 16542 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:04:13.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:13.902585 16542 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:04:13.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:13.911191 16542 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:04:13.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:13.911407 16542 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.243.1
02:04:13.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:13.921241 16542 tablet_server_runner.cc:78] Tablet server non-default flags:
02:04:13.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:04:13.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.13.243.20:34675
02:04:13.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:04:13.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:04:13.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-0/data
02:04:13.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-0/wal
02:04:13.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:04:13.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:04:13.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.13.243.1:33421
02:04:13.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:04:13.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:04:13.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:04:13.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-0/data/info.pb
02:04:13.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.13.243.1
02:04:13.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42315
02:04:13.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.13.243.62:45883,127.13.243.61:44961,127.13.243.60:35015
02:04:13.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:04:13.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:04:13.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:04:13.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:04:13.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:04:13.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.13.243.1
02:04:13.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-0/logs
02:04:13.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:04:13.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:04:13.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:04:13.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:04:13.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:04:13.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision a3f35afc2b6dad6aa403ec58378221544a26edd0
02:04:13.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:04:13.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:40:41 UTC on bdcb31816ec0
02:04:13.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 16849
02:04:13.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
02:04:13.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:13.923354 16542 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:04:13.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:13.925137 16542 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:04:13.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:13.935286 16550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:04:13.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:13.935359 16549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:04:13.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:13.936323 16552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:04:13.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:13.936581 16542 server_base.cc:1061] running on GCE node
02:04:13.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:13.937167 16542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:04:13.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:13.938169 16542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:04:13.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:13.939386 16542 hybrid_clock.cc:648] HybridClock initialized: now 1783476253939340 us; error 70 us; skew 500 ppm
02:04:13.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:13.939723 16542 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:04:13.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:13.941869 16542 webserver.cc:534] Webserver started at http://127.13.243.1:42315/ using document root <none> and password file <none>
02:04:13.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:13.942660 16542 fs_manager.cc:362] Metadata directory not provided
02:04:13.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:13.942788 16542 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:04:13.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:13.948158 16542 fs_manager.cc:714] Time spent opening directory manager: real 0.003s	user 0.005s	sys 0.000s
02:04:13.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:13.950907 16558 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:04:13.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:13.952169 16542 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
02:04:13.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:13.952358 16542 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-0/data,/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-0/wal
02:04:13.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "6422ba2afbf742febb59b258c6ac5101"
02:04:13.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:03:36 on dist-test-slave-pjt5"
02:04:13.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:13.953053 16542 fs_report.cc:389] FS layout report
02:04:13.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:04:13.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-0/wal
02:04:13.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-0/wal
02:04:13.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-0/data/data
02:04:13.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:04:13.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:04:13.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:04:13.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:04:13.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:04:13.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:04:13.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:04:13.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:04:13.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:04:13.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:04:13.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:04:14.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:14.005780 16542 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:04:14.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:14.006839 16542 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:04:14.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:14.007179 16542 kserver.cc:163] Server-wide thread pool size limit: 3276
02:04:14.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:14.008203 16542 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:04:14.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:14.010305 16565 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
02:04:14.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:14.016238 16542 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
02:04:14.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:14.016372 16542 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.006s	user 0.001s	sys 0.000s
02:04:14.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:14.016516 16542 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
02:04:14.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:14.019594 16542 ts_tablet_manager.cc:616] Registered 1 tablets
02:04:14.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:14.019721 16542 ts_tablet_manager.cc:595] Time spent register tablets: real 0.003s	user 0.002s	sys 0.000s
02:04:14.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:14.020046 16565 tablet_bootstrap.cc:492] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101: Bootstrap starting.
02:04:14.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:14.048363 16565 log.cc:826] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101: Log is configured to *not* fsync() on all Append() calls
02:04:14.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:14.067735 16542 rpc_server.cc:307] RPC server started. Bound to: 127.13.243.1:33421
02:04:14.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:14.067797 16672 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.243.1:33421 every 8 connection(s)
02:04:14.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:14.069170 16542 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-0/data/info.pb
02:04:14.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:14.074146 14284 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu as pid 16542
02:04:14.074 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:04:14.108 [DEBUG - kudu-nio-3] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 30 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291, tablet=89c23a195e1e46c19d1cff4420b1f6e5, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:04:14.110 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.13.243.3:41985
02:04:14.110 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

02:04:14.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:14.111035 14284 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu pid 16180
02:04:14.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:14.134272 16565 tablet_bootstrap.cc:492] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101: Bootstrap replayed 1/1 log segments. Stats: ops{read=36 overwritten=0 applied=36 ignored=0} inserts{seen=27 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
02:04:14.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:14.135080 16565 tablet_bootstrap.cc:492] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101: Bootstrap complete.
02:04:14.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:14.136886 16565 ts_tablet_manager.cc:1403] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101: Time spent bootstrapping tablet: real 0.117s	user 0.071s	sys 0.005s
02:04:14.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:14.145385 16673 heartbeater.cc:344] Connected to a master server at 127.13.243.60:35015
02:04:14.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:14.145768 16673 heartbeater.cc:461] Registering TS with master...
02:04:14.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:14.146831 16673 heartbeater.cc:507] Master 127.13.243.60:35015 requested a full tablet report, sending...
02:04:14.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:14.149770 14468 ts_manager.cc:194] Re-registered known tserver with Master: 6422ba2afbf742febb59b258c6ac5101 (127.13.243.1:33421)
02:04:14.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:14.151283 14468 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.13.243.1:39027
02:04:14.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:14.157120 16565 raft_consensus.cc:359] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [term 9 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:04:14.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:14.158015 16674 heartbeater.cc:344] Connected to a master server at 127.13.243.62:45883
02:04:14.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:14.158169 16674 heartbeater.cc:461] Registering TS with master...
02:04:14.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:14.158141 16565 raft_consensus.cc:740] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [term 9 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6422ba2afbf742febb59b258c6ac5101, State: Initialized, Role: FOLLOWER
02:04:14.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:14.158635 16674 heartbeater.cc:507] Master 127.13.243.62:45883 requested a full tablet report, sending...
02:04:14.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:14.164614 16565 consensus_queue.cc:260] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 36, Last appended: 9.36, Last appended by leader: 36, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:04:14.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:14.167173 14326 ts_manager.cc:194] Re-registered known tserver with Master: 6422ba2afbf742febb59b258c6ac5101 (127.13.243.1:33421)
02:04:14.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:14.170413 16675 heartbeater.cc:344] Connected to a master server at 127.13.243.61:44961
02:04:14.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:14.170565 16675 heartbeater.cc:461] Registering TS with master...
02:04:14.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:14.170866 16675 heartbeater.cc:507] Master 127.13.243.61:44961 requested a full tablet report, sending...
02:04:14.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:14.172633 14397 ts_manager.cc:194] Re-registered known tserver with Master: 6422ba2afbf742febb59b258c6ac5101 (127.13.243.1:33421)
02:04:14.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:14.173072 16673 heartbeater.cc:499] Master 127.13.243.60:35015 was elected leader, sending a full tablet report...
02:04:14.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:14.173862 16565 ts_tablet_manager.cc:1434] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101: Time spent starting tablet: real 0.037s	user 0.013s	sys 0.000s
02:04:14.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:14.181313 16306 generic_service.cc:196] Checking for leaks (request via RPC)
02:04:14.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:14.281242 14284 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu with pid 16180
02:04:14.301 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:04:14.301 [DEBUG - kudu-nio-3] (Connection.java:239) [peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)] handling channelInactive
02:04:14.302 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)] cleaning up while in state READY due to: connection closed
02:04:14.302 [DEBUG - Time-limited test] (Connection.java:748) [peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)] connecting to peer
02:04:14.303 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:04:14.303 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:04:14.304 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985) for tablet 89c23a195e1e46c19d1cff4420b1f6e5: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:04:14.327 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:04:14.327 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421),7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)[L],b123247401d540528f8324235e936260(127.13.243.2:38235)]}]
02:04:14.328 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=89c23a195e1e46c19d1cff4420b1f6e5, attempt=2, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=26), Traces: [0ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [1ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [2ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [22ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [25ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=Deferred@97962348(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f87c036bed0@351785cd -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3ac20257 -> wakeup thread Time-limited test)) after lookup
02:04:14.328 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)] connecting to peer
02:04:14.329 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:04:14.329 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:04:14.329 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985) for tablet 89c23a195e1e46c19d1cff4420b1f6e5: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:04:14.347 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:04:14.347 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421),7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)[L],b123247401d540528f8324235e936260(127.13.243.2:38235)]}]
02:04:14.348 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=89c23a195e1e46c19d1cff4420b1f6e5, attempt=4, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=46), Traces: [0ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [1ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [2ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [22ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [25ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [26ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [27ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [27ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [42ms] refreshing cache from master, [42ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [45ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=Deferred@97962348(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f87c036bed0@351785cd -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@663486586 after retry RPC@234258022), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3ac20257 -> wakeup thread Time-limited test -> (continuation of Deferred@663486586 after retry RPC@234258022))) after lookup
02:04:14.348 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)] connecting to peer
02:04:14.349 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:04:14.349 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:04:14.349 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985) for tablet 89c23a195e1e46c19d1cff4420b1f6e5: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:04:14.387 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:04:14.388 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421),7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)[L],b123247401d540528f8324235e936260(127.13.243.2:38235)]}]
02:04:14.388 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=89c23a195e1e46c19d1cff4420b1f6e5, attempt=6, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=86), Traces: [0ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [1ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [2ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [22ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [25ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [26ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [27ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [27ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [42ms] refreshing cache from master, [42ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [45ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [46ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [47ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [47ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [82ms] refreshing cache from master, [82ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [85ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=Deferred@97962348(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f87c036bed0@351785cd -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@663486586 after retry RPC@234258022) -> (continuation of Deferred@149529477 after retry RPC@911407536), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3ac20257 -> wakeup thread Time-limited test -> (continuation of Deferred@663486586 after retry RPC@234258022) -> (continuation of Deferred@149529477 after retry RPC@911407536))) after lookup
02:04:14.389 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)] connecting to peer
02:04:14.390 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:04:14.390 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:04:14.390 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985) for tablet 89c23a195e1e46c19d1cff4420b1f6e5: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:04:14.427 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:04:14.428 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421),7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)[L],b123247401d540528f8324235e936260(127.13.243.2:38235)]}]
02:04:14.428 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=89c23a195e1e46c19d1cff4420b1f6e5, attempt=8, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=126), Traces: [0ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [1ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [2ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [22ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [25ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [26ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [27ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [27ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [42ms] refreshing cache from master, [42ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [45ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [46ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [47ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [47ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [82ms] refreshing cache from master, [82ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [85ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [86ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [88ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [88ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [122ms] refreshing cache from master, [122ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [125ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=Deferred@97962348(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f87c036bed0@351785cd -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@663486586 after retry RPC@234258022) -> (continuation of Deferred@149529477 after retry RPC@911407536) -> (continuation of Deferred@2145147459 after retry RPC@1591911281), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3ac20257 -> wakeup thread Time-limited test -> (continuation of Deferred@663486586 after retry RPC@234258022) -> (continuation of Deferred@149529477 after retry RPC@911407536) -> (continuation of Deferred@2145147459 after retry RPC@1591911281))) after lookup
02:04:14.429 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)] connecting to peer
02:04:14.430 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:04:14.430 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:04:14.430 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985) for tablet 89c23a195e1e46c19d1cff4420b1f6e5: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:04:14.767 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:04:14.767 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421),7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)[L],b123247401d540528f8324235e936260(127.13.243.2:38235)]}]
02:04:14.768 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=89c23a195e1e46c19d1cff4420b1f6e5, attempt=10, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=466), Traces: [0ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [1ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [2ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [22ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [25ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [26ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [27ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [27ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [42ms] refreshing cache from master, [42ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [45ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [46ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [47ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [47ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [82ms] refreshing cache from master, [82ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [85ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [86ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [88ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [88ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [122ms] refreshing cache from master, [122ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [125ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [126ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [128ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [128ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [462ms] refreshing cache from master, [462ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [465ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=Deferred@97962348(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f87c036bed0@351785cd -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@663486586 after retry RPC@234258022) -> (continuation of Deferred@149529477 after retry RPC@911407536) -> (continuation of Deferred@2145147459 after retry RPC@1591911281) -> (continuation of Deferred@2062217286 after retry RPC@827862337), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3ac20257 -> wakeup thread Time-limited test -> (continuation of Deferred@663486586 after retry RPC@234258022) -> (continuation of Deferred@149529477 after retry RPC@911407536) -> (continuation of Deferred@2145147459 after retry RPC@1591911281) -> (continuation of Deferred@2062217286 after retry RPC@827862337))) after lookup
02:04:14.768 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)] connecting to peer
02:04:14.769 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:04:14.769 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:04:14.769 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985) for tablet 89c23a195e1e46c19d1cff4420b1f6e5: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985
02:04:15.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:15.466185 16688 raft_consensus.cc:493] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 7f08c11f6dc94dc3aa69febd1051f2e9)
02:04:15.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:15.466470 16688 raft_consensus.cc:515] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:04:15.468 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:15.468142 16688 leader_election.cc:290] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6422ba2afbf742febb59b258c6ac5101 (127.13.243.1:33421), 7f08c11f6dc94dc3aa69febd1051f2e9 (127.13.243.3:41985)
02:04:15.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:15.470033 16373 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.243.3:41985: connect: Connection refused (error 111)
02:04:15.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:15.471808 16373 leader_election.cc:336] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 7f08c11f6dc94dc3aa69febd1051f2e9 (127.13.243.3:41985): Network error: Client connection negotiation failed: client connection to 127.13.243.3:41985: connect: Connection refused (error 111)
02:04:15.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:15.475047 16693 raft_consensus.cc:493] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
02:04:15.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:15.475296 16693 raft_consensus.cc:515] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:04:15.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:15.476910 16693 leader_election.cc:290] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b123247401d540528f8324235e936260 (127.13.243.2:38235), 7f08c11f6dc94dc3aa69febd1051f2e9 (127.13.243.3:41985)
02:04:15.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:15.477306 16627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89c23a195e1e46c19d1cff4420b1f6e5" candidate_uuid: "b123247401d540528f8324235e936260" candidate_term: 11 candidate_status { last_received { term: 10 index: 40 } } ignore_live_leader: false dest_uuid: "6422ba2afbf742febb59b258c6ac5101" is_pre_election: true
02:04:15.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:15.477842 16627 raft_consensus.cc:2468] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b123247401d540528f8324235e936260 in term 9.
02:04:15.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:15.479489 16373 leader_election.cc:304] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6422ba2afbf742febb59b258c6ac5101, b123247401d540528f8324235e936260; no voters: 7f08c11f6dc94dc3aa69febd1051f2e9
02:04:15.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:15.480297 16688 raft_consensus.cc:2804] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [term 10 FOLLOWER]: Leader pre-election won for term 11
02:04:15.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:15.480439 16688 raft_consensus.cc:493] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 7f08c11f6dc94dc3aa69febd1051f2e9)
02:04:15.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:15.480548 16688 raft_consensus.cc:3060] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [term 10 FOLLOWER]: Advancing to term 11
02:04:15.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:15.484869 16688 raft_consensus.cc:515] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:04:15.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:15.486382 16627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89c23a195e1e46c19d1cff4420b1f6e5" candidate_uuid: "b123247401d540528f8324235e936260" candidate_term: 11 candidate_status { last_received { term: 10 index: 40 } } ignore_live_leader: false dest_uuid: "6422ba2afbf742febb59b258c6ac5101"
02:04:15.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:15.486627 16627 raft_consensus.cc:3060] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [term 9 FOLLOWER]: Advancing to term 11
02:04:15.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:15.494669 16627 raft_consensus.cc:2468] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b123247401d540528f8324235e936260 in term 11.
02:04:15.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:15.495174 16688 leader_election.cc:290] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [CANDIDATE]: Term 11 election: Requested vote from peers 6422ba2afbf742febb59b258c6ac5101 (127.13.243.1:33421), 7f08c11f6dc94dc3aa69febd1051f2e9 (127.13.243.3:41985)
02:04:15.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:15.495298 16373 leader_election.cc:304] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6422ba2afbf742febb59b258c6ac5101, b123247401d540528f8324235e936260; no voters: 
02:04:15.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:15.494697 16559 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.243.3:41985: connect: Connection refused (error 111)
02:04:15.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:15.496026 16688 raft_consensus.cc:2804] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [term 11 FOLLOWER]: Leader election won for term 11
02:04:15.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:15.497521 16373 leader_election.cc:336] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 7f08c11f6dc94dc3aa69febd1051f2e9 (127.13.243.3:41985): Network error: Client connection negotiation failed: client connection to 127.13.243.3:41985: connect: Connection refused (error 111)
02:04:15.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:15.499050 16688 raft_consensus.cc:697] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [term 11 LEADER]: Becoming Leader. State: Replica: b123247401d540528f8324235e936260, State: Running, Role: LEADER
02:04:15.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:15.499607 16688 consensus_queue.cc:237] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 10.40, Last appended by leader: 40, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:04:15.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:15.504452 16442 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89c23a195e1e46c19d1cff4420b1f6e5" candidate_uuid: "6422ba2afbf742febb59b258c6ac5101" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "b123247401d540528f8324235e936260" is_pre_election: true
02:04:15.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:15.507685 14468 catalog_manager.cc:5696] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 reported cstate change: term changed from 10 to 11, leader changed from 7f08c11f6dc94dc3aa69febd1051f2e9 (127.13.243.3) to b123247401d540528f8324235e936260 (127.13.243.2). New cstate: current_term: 11 leader_uuid: "b123247401d540528f8324235e936260" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } health_report { overall_health: UNKNOWN } } }
02:04:15.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:15.509186 16559 leader_election.cc:336] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 7f08c11f6dc94dc3aa69febd1051f2e9 (127.13.243.3:41985): Network error: Client connection negotiation failed: client connection to 127.13.243.3:41985: connect: Connection refused (error 111)
02:04:15.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:15.509409 16559 leader_election.cc:304] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6422ba2afbf742febb59b258c6ac5101; no voters: 7f08c11f6dc94dc3aa69febd1051f2e9, b123247401d540528f8324235e936260
02:04:15.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:15.509840 16693 raft_consensus.cc:2749] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [term 11 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
02:04:15.708 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 89c23a195e1e46c19d1cff4420b1f6e5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291' with partition [<start>, <end>)
02:04:15.708 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=89c23a195e1e46c19d1cff4420b1f6e5@[6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421),7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985),b123247401d540528f8324235e936260(127.13.243.2:38235)[L]]}]
02:04:15.708 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=89c23a195e1e46c19d1cff4420b1f6e5, attempt=12, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=1406), Traces: [0ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [1ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [2ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [22ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [25ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [26ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [27ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [27ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [42ms] refreshing cache from master, [42ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [45ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [46ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [47ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [47ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [82ms] refreshing cache from master, [82ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [85ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [86ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [88ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [88ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [122ms] refreshing cache from master, [122ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [125ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [126ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [128ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [128ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [462ms] refreshing cache from master, [462ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [465ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, [466ms] sending RPC to server 7f08c11f6dc94dc3aa69febd1051f2e9, [467ms] received response from server 7f08c11f6dc94dc3aa69febd1051f2e9: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [467ms] delaying RPC due to: Network error: Failed to connect to peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985): Connection refused: /127.13.243.3:41985, [1402ms] refreshing cache from master, [1402ms] Sub RPC GetTableLocations: sending RPC to server master-127.13.243.60:35015, [1405ms] Sub RPC GetTableLocations: received response from server master-127.13.243.60:35015: OK, deferred=Deferred@97962348(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f87c036bed0@351785cd -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@663486586 after retry RPC@234258022) -> (continuation of Deferred@149529477 after retry RPC@911407536) -> (continuation of Deferred@2145147459 after retry RPC@1591911281) -> (continuation of Deferred@2062217286 after retry RPC@827862337) -> (continuation of Deferred@2070915553 after retry RPC@1472264005), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3ac20257 -> wakeup thread Time-limited test -> (continuation of Deferred@663486586 after retry RPC@234258022) -> (continuation of Deferred@149529477 after retry RPC@911407536) -> (continuation of Deferred@2145147459 after retry RPC@1591911281) -> (continuation of Deferred@2062217286 after retry RPC@827862337) -> (continuation of Deferred@2070915553 after retry RPC@1472264005))) after lookup
02:04:15.709 [DEBUG - kudu-nio-1] (Connection.java:748) [peer b123247401d540528f8324235e936260(127.13.243.2:38235)] connecting to peer
02:04:15.710 [DEBUG - kudu-nio-2] (Connection.java:757) [peer b123247401d540528f8324235e936260(127.13.243.2:38235)] Successfully connected to peer
02:04:15.729 [DEBUG - kudu-nio-2] (Negotiator.java:919) Authenticated connection [id: 0xf48bf50b, L:/127.0.0.1:54414 - R:/127.13.243.2:38235] using TOKEN/null
02:04:15.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:15.735416 16627 raft_consensus.cc:1275] T 89c23a195e1e46c19d1cff4420b1f6e5 P 6422ba2afbf742febb59b258c6ac5101 [term 11 FOLLOWER]: Refusing update from remote peer b123247401d540528f8324235e936260: Log matching property violated. Preceding OpId in replica: term: 9 index: 36. Preceding OpId from leader: term: 11 index: 42. (index mismatch)
02:04:15.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:15.736004 16373 consensus_peers.cc:597] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 -> Peer 7f08c11f6dc94dc3aa69febd1051f2e9 (127.13.243.3:41985): Couldn't send request to peer 7f08c11f6dc94dc3aa69febd1051f2e9. Status: Network error: Client connection negotiation failed: client connection to 127.13.243.3:41985: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
02:04:15.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:15.736609 16688 consensus_queue.cc:1048] T 89c23a195e1e46c19d1cff4420b1f6e5 P b123247401d540528f8324235e936260 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 36, Time since last communication: 0.000s
02:04:15.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:15.750068 16712 mvcc.cc:204] Tried to move back new op lower bound from 7305118743481057280 to 7305118742531002368. Current Snapshot: MvccSnapshot[applied={T|T < 7305118743481057280}]
02:04:15.772 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.13.243.3:41985
02:04:15.772 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

02:04:15.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:15.773154 14284 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu
02:04:15.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu
02:04:15.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/wal
02:04:15.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/data
02:04:15.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:04:15.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:04:15.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:04:15.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:04:15.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:04:15.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:04:15.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/logs
02:04:15.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/data/info.pb
02:04:15.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:04:15.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:04:15.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:04:15.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:04:15.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:04:15.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:04:15.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:04:15.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:04:15.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:04:15.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:04:15.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.13.243.3:41985
02:04:15.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.13.243.3
02:04:15.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.13.243.62:45883,127.13.243.61:44961,127.13.243.60:35015
02:04:15.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=43625
02:04:15.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.13.243.3
02:04:15.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.13.243.20:34675
02:04:15.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:04:15.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:04:15.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:04:16.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:16.144119 16720 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:04:16.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:16.144479 16720 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:04:16.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:16.144596 16720 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:04:16.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:16.153748 16720 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:04:16.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:16.153973 16720 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.243.3
02:04:16.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.164880 16720 tablet_server_runner.cc:78] Tablet server non-default flags:
02:04:16.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:04:16.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.13.243.20:34675
02:04:16.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:04:16.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:04:16.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/data
02:04:16.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/wal
02:04:16.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:04:16.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:04:16.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.13.243.3:41985
02:04:16.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:04:16.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:04:16.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:04:16.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/data/info.pb
02:04:16.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.13.243.3
02:04:16.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=43625
02:04:16.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.13.243.62:45883,127.13.243.61:44961,127.13.243.60:35015
02:04:16.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:04:16.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:04:16.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:04:16.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:04:16.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:04:16.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.13.243.3
02:04:16.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/logs
02:04:16.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:04:16.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:04:16.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:04:16.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:04:16.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:04:16.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision a3f35afc2b6dad6aa403ec58378221544a26edd0
02:04:16.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:04:16.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:40:41 UTC on bdcb31816ec0
02:04:16.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 16849
02:04:16.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
02:04:16.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.166954 16720 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:04:16.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.168812 16720 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:04:16.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:16.178865 16727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:04:16.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:16.179368 16726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:04:16.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:16.179759 16729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:04:16.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.180207 16720 server_base.cc:1061] running on GCE node
02:04:16.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.180747 16720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:04:16.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.181674 16720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:04:16.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.182912 16720 hybrid_clock.cc:648] HybridClock initialized: now 1783476256182875 us; error 46 us; skew 500 ppm
02:04:16.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.183277 16720 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:04:16.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.185555 16720 webserver.cc:534] Webserver started at http://127.13.243.3:43625/ using document root <none> and password file <none>
02:04:16.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.186408 16720 fs_manager.cc:362] Metadata directory not provided
02:04:16.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.186542 16720 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:04:16.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.192328 16720 fs_manager.cc:714] Time spent opening directory manager: real 0.004s	user 0.005s	sys 0.000s
02:04:16.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.195364 16735 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:04:16.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.196707 16720 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
02:04:16.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.196938 16720 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/data,/tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/wal
02:04:16.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "7f08c11f6dc94dc3aa69febd1051f2e9"
02:04:16.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:03:37 on dist-test-slave-pjt5"
02:04:16.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.197743 16720 fs_report.cc:389] FS layout report
02:04:16.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:04:16.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/wal
02:04:16.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/wal
02:04:16.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/data/data
02:04:16.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:04:16.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:04:16.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:04:16.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:04:16.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:04:16.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:04:16.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:04:16.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:04:16.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:04:16.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:04:16.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:04:16.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.224148 16720 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:04:16.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.225139 16720 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:04:16.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.225500 16720 kserver.cc:163] Server-wide thread pool size limit: 3276
02:04:16.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.226676 16720 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:04:16.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.229079 16742 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
02:04:16.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.235245 16720 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
02:04:16.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.235381 16720 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.007s	user 0.001s	sys 0.000s
02:04:16.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.235524 16720 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
02:04:16.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.238799 16720 ts_tablet_manager.cc:616] Registered 1 tablets
02:04:16.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.238894 16720 ts_tablet_manager.cc:595] Time spent register tablets: real 0.003s	user 0.003s	sys 0.000s
02:04:16.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.239395 16742 tablet_bootstrap.cc:492] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9: Bootstrap starting.
02:04:16.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.270169 16742 log.cc:826] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9: Log is configured to *not* fsync() on all Append() calls
02:04:16.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.307987 16720 rpc_server.cc:307] RPC server started. Bound to: 127.13.243.3:41985
02:04:16.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.308036 16851 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.243.3:41985 every 8 connection(s)
02:04:16.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.309492 16720 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskRwVx6v/test-tmp/mini-kudu-cluster15650486011252422638/ts-2/data/info.pb
02:04:16.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.314886 16742 tablet_bootstrap.cc:492] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9: Bootstrap replayed 1/1 log segments. Stats: ops{read=40 overwritten=0 applied=40 ignored=0} inserts{seen=30 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
02:04:16.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.314930 14284 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu as pid 16720
02:04:16.315 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:04:16.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.315570 16742 tablet_bootstrap.cc:492] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9: Bootstrap complete.
02:04:16.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.317142 16742 ts_tablet_manager.cc:1403] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9: Time spent bootstrapping tablet: real 0.078s	user 0.052s	sys 0.021s
02:04:16.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.326175 16742 raft_consensus.cc:359] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [term 10 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:04:16.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.327296 16742 raft_consensus.cc:740] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [term 10 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f08c11f6dc94dc3aa69febd1051f2e9, State: Initialized, Role: FOLLOWER
02:04:16.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.328100 16742 consensus_queue.cc:260] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 40, Last appended: 10.40, Last appended by leader: 40, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6422ba2afbf742febb59b258c6ac5101" member_type: VOTER last_known_addr { host: "127.13.243.1" port: 33421 } } peers { permanent_uuid: "b123247401d540528f8324235e936260" member_type: VOTER last_known_addr { host: "127.13.243.2" port: 38235 } } peers { permanent_uuid: "7f08c11f6dc94dc3aa69febd1051f2e9" member_type: VOTER last_known_addr { host: "127.13.243.3" port: 41985 } }
02:04:16.334 [DEBUG - kudu-nio-2] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 33 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291, tablet=89c23a195e1e46c19d1cff4420b1f6e5, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:04:16.341 [DEBUG - kudu-nio-2] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 33 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783476218291, tablet=89c23a195e1e46c19d1cff4420b1f6e5, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:04:16.350 [DEBUG - main] (Connection.java:575) [peer master-127.13.243.61:44961(127.13.243.61:44961)] disconnecting while in state READY
02:04:16.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.350028 16852 heartbeater.cc:344] Connected to a master server at 127.13.243.60:35015
02:04:16.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.350528 16852 heartbeater.cc:461] Registering TS with master...
02:04:16.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.352191 16852 heartbeater.cc:507] Master 127.13.243.60:35015 requested a full tablet report, sending...
02:04:16.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.355603 14468 ts_manager.cc:194] Re-registered known tserver with Master: 7f08c11f6dc94dc3aa69febd1051f2e9 (127.13.243.3:41985)
02:04:16.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.357147 14468 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.13.243.3:60013
02:04:16.358 [DEBUG - kudu-nio-2] (Connection.java:239) [peer master-127.13.243.61:44961(127.13.243.61:44961)] handling channelInactive
02:04:16.359 [DEBUG - main] (Connection.java:575) [peer 6422ba2afbf742febb59b258c6ac5101(127.13.243.1:33421)] disconnecting while in state TERMINATED
02:04:16.359 [DEBUG - main] (Connection.java:575) [peer b123247401d540528f8324235e936260(127.13.243.2:38235)] disconnecting while in state READY
02:04:16.359 [DEBUG - main] (Connection.java:575) [peer master-127.13.243.62:45883(127.13.243.62:45883)] disconnecting while in state READY
02:04:16.359 [DEBUG - kudu-nio-2] (Connection.java:709) [peer master-127.13.243.61:44961(127.13.243.61:44961)] cleaning up while in state READY due to: connection closed
02:04:16.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.357843 16853 heartbeater.cc:344] Connected to a master server at 127.13.243.62:45883
02:04:16.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.358052 16853 heartbeater.cc:461] Registering TS with master...
02:04:16.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.358834 16853 heartbeater.cc:507] Master 127.13.243.62:45883 requested a full tablet report, sending...
02:04:16.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.360939 14326 ts_manager.cc:194] Re-registered known tserver with Master: 7f08c11f6dc94dc3aa69febd1051f2e9 (127.13.243.3:41985)
02:04:16.363 [DEBUG - kudu-nio-2] (Connection.java:239) [peer b123247401d540528f8324235e936260(127.13.243.2:38235)] handling channelInactive
02:04:16.363 [DEBUG - kudu-nio-2] (Connection.java:709) [peer b123247401d540528f8324235e936260(127.13.243.2:38235)] cleaning up while in state READY due to: connection closed
02:04:16.364 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.13.243.62:45883(127.13.243.62:45883)] handling channelInactive
02:04:16.364 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.13.243.62:45883(127.13.243.62:45883)] cleaning up while in state READY due to: connection closed
02:04:16.365 [DEBUG - main] (Connection.java:575) [peer 7f08c11f6dc94dc3aa69febd1051f2e9(127.13.243.3:41985)] disconnecting while in state TERMINATED
02:04:16.365 [DEBUG - main] (Connection.java:575) [peer master-127.13.243.60:35015(127.13.243.60:35015)] disconnecting while in state READY
02:04:16.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.365621 16854 heartbeater.cc:344] Connected to a master server at 127.13.243.61:44961
02:04:16.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.365829 16854 heartbeater.cc:461] Registering TS with master...
02:04:16.366 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.13.243.60:35015(127.13.243.60:35015)] handling channelInactive
02:04:16.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.366336 16854 heartbeater.cc:507] Master 127.13.243.61:44961 requested a full tablet report, sending...
02:04:16.366 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.13.243.60:35015(127.13.243.60:35015)] cleaning up while in state READY due to: connection closed
02:04:16.367 [DEBUG - main] (AsyncKuduClient.java:2704) Releasing all remaining resources
02:04:16.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.368076 14397 ts_manager.cc:194] Re-registered known tserver with Master: 7f08c11f6dc94dc3aa69febd1051f2e9 (127.13.243.3:41985)
02:04:16.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.370528 14284 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu pid 16542
02:04:16.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.372722 16852 heartbeater.cc:499] Master 127.13.243.60:35015 was elected leader, sending a full tablet report...
02:04:16.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.373361 16742 ts_tablet_manager.cc:1434] T 89c23a195e1e46c19d1cff4420b1f6e5 P 7f08c11f6dc94dc3aa69febd1051f2e9: Time spent starting tablet: real 0.056s	user 0.024s	sys 0.028s
02:04:16.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.389933 16668 generic_service.cc:196] Checking for leaks (request via RPC)
02:04:16.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.483948 14284 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu with pid 16542
02:04:16.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.502633 14284 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu pid 16358
02:04:16.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.511283 16482 generic_service.cc:196] Checking for leaks (request via RPC)
02:04:16.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.624873 14284 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu with pid 16358
02:04:16.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.640717 14284 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu pid 16720
02:04:16.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.649017 16847 generic_service.cc:196] Checking for leaks (request via RPC)
02:04:16.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.739511 14284 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu with pid 16720
02:04:16.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.757146 14284 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu pid 14295
02:04:16.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.758491 14356 generic_service.cc:196] Checking for leaks (request via RPC)
02:04:16.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.877188 14284 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu with pid 14295
02:04:16.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.900386 14284 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu pid 14363
02:04:16.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:16.901479 14427 generic_service.cc:196] Checking for leaks (request via RPC)
02:04:17.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:17.017982 14284 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu with pid 14363
02:04:17.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:17.042078 14454 connection.cc:573] client connection to 127.13.243.61:44961 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
02:04:17.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:04:17.042397 14454 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
02:04:17.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:17.042506 14284 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu pid 14434
02:04:17.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:17.043665 14498 generic_service.cc:196] Checking for leaks (request via RPC)
02:04:17.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:04:17.167817 14284 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskRwVx6v/build/asan/bin/kudu with pid 14434
02:04:17.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-07-08T02:04:17Z chronyd exiting

Time: 43.661

OK (1 test)

